recipe-maintainers/cc-ci
0
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Merge branch 'restructure/recipe-custom' — recipe-customization restructure (rcust M1 PASS @858e0f5, REVIEW-rcust 01f9f70)
All checks were successful
continuous-integration/drone/push Build is passing
Details

Browse Source 
Single registry-backed meta loader, legacy key/path deletion, uniform ctx hooks, custom-test
placement rule + fixtures, customization manifest, docs. M2 real-CI regression sweep follows.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
autonomic-bot
2026-06-10 19:28:38 +00:00
parent 01f9f70970 858e0f582f
commit 01e6d497ba
66 changed files with 2040 additions and 1097 deletions
Download Patch File Download Diff File Expand all files Collapse all files

104
docs/enroll-recipe.md
View File

@ -14,8 +14,9 @@ those are discovered and run against the live app (D4 — see below).
``` ```
tests/<recipe>/ tests/<recipe>/
├── recipe_meta.py # optional per-recipe harness config (see below) ├── recipe_meta.py # optional per-recipe harness config (see below)
├── install_steps.sh # optional custom install-steps hook (pre-deploy setup) ├── install_steps.sh # optional custom install-steps hook (pre-deploy setup + deps env wiring)
├── ops.py # optional pre-op seed hooks (pre_install/pre_upgrade/pre_backup/pre_restore) ├── compose.ccci.yml # optional CI-only compose overlay (harness-copied, auto-chaos base deploy)
├── ops.py # optional pre_<op>(ctx) seed hooks (install/upgrade/backup/restore)
├── test_install.py # optional install overlay (runs ADDITIVELY alongside generic) ├── test_install.py # optional install overlay (runs ADDITIVELY alongside generic)
├── test_upgrade.py # optional upgrade overlay (runs ADDITIVELY alongside generic) ├── test_upgrade.py # optional upgrade overlay (runs ADDITIVELY alongside generic)
├── test_backup.py # optional backup overlay (runs ADDITIVELY alongside generic) ├── test_backup.py # optional backup overlay (runs ADDITIVELY alongside generic)
@ -39,11 +40,14 @@ To add recipe-specific coverage, drop a `tests/<recipe>/test_<op>.py` **overlay*
**ALONGSIDE** the generic for that op (HC3 additive, Phase 1e); the generic floor is never silently **ALONGSIDE** the generic for that op (HC3 additive, Phase 1e); the generic floor is never silently
dropped. Overlays are **assertion-only** against the shared live deployment (the `live_app` fixture; dropped. Overlays are **assertion-only** against the shared live deployment (the `live_app` fixture;
they never perform the op or deploy/teardown — the orchestrator owns those). If the overlay needs to they never perform the op or deploy/teardown — the orchestrator owns those). If the overlay needs to
SEED pre-op state (data-continuity markers, the backup→restore divergence), put `pre_<op>(domain, SEED pre-op state (data-continuity markers, the backup→restore divergence), put `pre_<op>(ctx)`
meta)` callables in `tests/<recipe>/ops.py` — the orchestrator runs them BEFORE the op. Copy an callables in `tests/<recipe>/ops.py` — the orchestrator runs them BEFORE the op (`ctx` is the
uniform `HookCtx` every hook receives — `docs/recipe-customization.md` §4.1). Copy an
existing recipe (`tests/custom-html/` simple/volume marker; `tests/keycloak/` admin-API; `tests/ existing recipe (`tests/custom-html/` simple/volume marker; `tests/keycloak/` admin-API; `tests/
matrix-synapse/` `db`-service psql marker). **Do not edit the shared `tests/conftest.py` / matrix-synapse/` `db`-service psql marker). **Do not edit the shared `tests/conftest.py` /
`runner/harness/` to add a recipe** — set per-recipe knobs in `recipe_meta.py`: `runner/harness/` to add a recipe** — set per-recipe knobs in `recipe_meta.py` (the COMPLETE key
reference is the generated table in `docs/recipe-customization.md` §4; unknown ALL-CAPS keys are
hard errors, recipe-private constants are underscore-prefixed `_FOO`):
```python ```python
HEALTH_PATH = "/realms/master" # path that returns a healthy status (default "/") HEALTH_PATH = "/realms/master" # path that returns a healthy status (default "/")
@ -51,9 +55,7 @@ HEALTH_OK = (200,) # acceptable status codes (default 200/301/302)
DEPLOY_TIMEOUT = 600 # seconds for services to converge (default 600) DEPLOY_TIMEOUT = 600 # seconds for services to converge (default 600)
HTTP_TIMEOUT = 600 # seconds for the app to answer (default 300) HTTP_TIMEOUT = 600 # seconds for the app to answer (default 300)
BACKUP_CAPABLE = True # override backup-capability auto-detect (default: scan compose) BACKUP_CAPABLE = True # override backup-capability auto-detect (default: scan compose)
EXTRA_ENV = {"KEY": "value"} # or EXTRA_ENV(domain) -> dict; extra .env keys set at deploy EXTRA_ENV = {"KEY": "value"} # or EXTRA_ENV(ctx) -> dict; extra .env keys set at deploy
SKIP_GENERIC = ["upgrade"] # per-recipe opt-out from the generic floor for the listed ops
# ("all"/"*" = every op); rarely needed — generic is the floor
``` ```
Useful `harness.lifecycle` helpers for overlays: `http_get`, `http_fetch`, `http_body`, Useful `harness.lifecycle` helpers for overlays: `http_get`, `http_fetch`, `http_body`,
@ -76,9 +78,10 @@ Beyond the lifecycle overlays, each recipe carries (plan §4.1):
- **`playwright/`** — browser flows where the recipe's core UX is a UI (P6). - **`playwright/`** — browser flows where the recipe's core UX is a UI (P6).
The orchestrator's **custom** tier discovers `test_*.py` in `tests/<recipe>/{functional,playwright}/` The orchestrator's **custom** tier discovers `test_*.py` in `tests/<recipe>/{functional,playwright}/`
(recursive, via `runner/harness/discovery.custom_tests`) and runs each as its own pytest against ONLY (the placement rule, via `runner/harness/discovery.custom_tests` — a top-level `test_*.py`
the same `live_app` shared deployment. Lifecycle-named files (`test_install.py`/etc.) are is a lifecycle overlay and nothing else) and runs each as its own pytest against the same
**excluded** from the custom tier — they live at the top level and run as lifecycle overlays. `live_app` shared deployment. Lifecycle-named files (`test_install.py`/etc.) are **excluded**
from the custom tier even inside those subdirs (safety net against double-running).
### 2.2 Recipe-test dependencies — DEPS = [...] (Phase 2 Q2.3) ### 2.2 Recipe-test dependencies — DEPS = [...] (Phase 2 Q2.3)
@ -89,23 +92,28 @@ them in `recipe_meta.py`:
DEPS = ["keycloak"] # one entry per dep recipe name (cc-ci tests/<dep>/ must exist + work) DEPS = ["keycloak"] # one entry per dep recipe name (cc-ci tests/<dep>/ must exist + work)
``` ```
The orchestrator (plan §4.2): The orchestrator (plan §4.2; install-time provisioning is the ONLY mode):
1. Reads `DEPS` BEFORE deploying the recipe under test. 1. Reads `DEPS` and provisions every dep **BEFORE the single deploy** of the recipe under test —
2. Deploys each dep at a per-run domain `<dep[:4]>-<6hex>.ci.commoninternet.net` (the 6hex is each dep at a per-run domain `<dep[:4]>-<6hex>.ci.commoninternet.net` (the 6hex is hashed from
hashed from `parent_recipe + pr + ref + dep_recipe` so two recipes' deps of the same kind do `parent_recipe + pr + ref + dep_recipe` so two recipes' deps of the same kind do not collide on
not collide on a single node). a single node), waited healthy using the dep's own `recipe_meta.py`.
3. Waits each dep healthy using its own `recipe_meta.py` (HEALTH_PATH/HEALTH_OK/timeouts). 2. Persists the full per-dep identity + SSO creds dict to `$CCCI_DEPS_FILE` (jq-readable JSON,
4. Persists `[{"recipe": "<dep>", "domain": "<dep-domain>"}, ...]` to `$CCCI_DEPS_FILE`. `{"<dep>": {"domain": ..., "realm": ..., "client_secret": ..., ...}}`).
5. Deploys + tests the recipe under test as usual. 3. Deploys the recipe under test — its `install_steps.sh` reads `$CCCI_DEPS_FILE` and wires
6. Tears down the dep LAST in `finally` (reverse declaration order, with `verify=True` — leaked OIDC env into that ONE deploy (no post-deploy redeploy). A dep-provisioning failure does NOT
block the run: the recipe deploys alone, generic tiers run, and `requires_deps` tests skip
with a counted reason (F2-11).
4. Tears down the dep LAST in `finally` (reverse declaration order, with `verify=True` — leaked
deps fail the run loudly per §9 teardown sacred / F2-5 fix). deps fail the run loudly per §9 teardown sacred / F2-5 fix).
Tests access dep domains via the **`deps_apps` pytest fixture** (`tests/conftest.py`): Tests access deps via the **`deps` pytest fixture** (`tests/conftest.py`) — entries expose
`.domain` plus the full creds dict (attribute or dict-style):
```python ```python
def test_my_recipe_uses_keycloak(live_app, deps_apps): @pytest.mark.requires_deps
assert "keycloak" in deps_apps, f"keycloak dep not deployed; {deps_apps}" def test_my_recipe_uses_keycloak(live_app, deps):
kc_domain = deps_apps["keycloak"] assert "keycloak" in deps, f"keycloak dep not deployed; {deps}"
kc_domain = deps["keycloak"].domain
… …
``` ```
@ -120,7 +128,7 @@ For OIDC-dependent recipes, the shared `runner/harness/sso.py` provides:
from harness import sso from harness import sso
creds = sso.setup_keycloak_realm( creds = sso.setup_keycloak_realm(
kc_domain, # = deps_apps["keycloak"] kc_domain, # = deps["keycloak"].domain
realm="my-realm", realm="my-realm",
client_id="my-client", client_id="my-client",
redirect_uris=[f"https://{live_app}/*"], redirect_uris=[f"https://{live_app}/*"],
@ -144,10 +152,10 @@ ARE provider-pluggable.
Not every recipe is a single HTTP app. `recipe_meta.py` + a few harness mechanisms cover the harder Not every recipe is a single HTTP app. `recipe_meta.py` + a few harness mechanisms cover the harder
shapes (proven on mumble, mailu, and the SSO-dependent suite): shapes (proven on mumble, mailu, and the SSO-dependent suite):
- **`EXTRA_ENV`** — a dict **or** a `callable(domain) -> dict`. The callable form derives values from - **`EXTRA_ENV`** — a dict **or** a `callable(ctx) -> dict`. The callable form derives values from
the per-run domain (e.g. `MAIL_DOMAIN`/`HOSTNAMES` for mailu, `SANDBOX_DOMAIN` for cryptpad). Applied the per-run domain (`ctx.domain` — e.g. `MAIL_DOMAIN`/`HOSTNAMES` for mailu, `SANDBOX_DOMAIN` for
at every deploy (`abra.env_set`), so a recipe enrolls with NO shared-harness change. cryptpad). Applied at every deploy (`abra.env_set`), so a recipe enrolls with NO shared-harness change.
- **`READY_PROBE(domain) -> [...]`** — readiness signals beyond replica-convergence + the app's - **`READY_PROBE(ctx) -> [...]`** — readiness signals beyond replica-convergence + the app's
`HEALTH_PATH`. Two probe shapes: `HEALTH_PATH`. Two probe shapes:
- HTTP: `{"host": "...", "path": "/...", "ok": (200,)}` (e.g. lasuite-drive collabora WOPI discovery). - HTTP: `{"host": "...", "path": "/...", "ok": (200,)}` (e.g. lasuite-drive collabora WOPI discovery).
- **TCP**: `{"tcp_host": "127.0.0.1", "tcp_port": 64738, "stable": 3}` — polls a socket connect N - **TCP**: `{"tcp_host": "127.0.0.1", "tcp_port": 64738, "stable": 3}` — polls a socket connect N
@ -155,16 +163,16 @@ shapes (proven on mumble, mailu, and the SSO-dependent suite):
service (mumble: the mumble-web sidecar serves HTTP 200 while the voice server on 64738 is still service (mumble: the mumble-web sidecar serves HTTP 200 while the voice server on 64738 is still
rebinding after an upgrade redeploy — the TCP probe gates the backup tier until the voice server is rebinding after an upgrade redeploy — the TCP probe gates the backup tier until the voice server is
actually up). Runs after install AND after the upgrade chaos redeploy. actually up). Runs after install AND after the upgrade chaos redeploy.
- **`CHAOS_BASE_DEPLOY = True`** — make the pinned base deploy use `--chaos` (skips abra's clean-tree + - **`compose.ccci.yml`** (first-class at `tests/<recipe>/compose.ccci.yml`) — a CI-only compose
lint gates, still deploys the explicitly-checked-out pinned version, NOT latest). Needed when an overlay the harness itself copies into the recipe checkout before the base deploy, automatically
`install_steps.sh` adds an UNTRACKED file to the recipe checkout (e.g. mumble copies a using `--chaos` for that deploy (the untracked file would otherwise trip abra's pinned-deploy
`compose.host-ports.yml` into versions that predate it) — abra's pinned-deploy clean-tree check would clean-tree check). Reference it from `EXTRA_ENV`'s `COMPOSE_FILE`. Minimal, justified fallback
otherwise FATA. `abra.recipe_checkout` force-checks-out (`-f`) so the upgrade tier's re-checkout to only (e.g. ghost's 15m `start_period` grace). `abra.recipe_checkout` force-checks-out (`-f`) so
PR-head overwrites such overlays cleanly. the upgrade tier's re-checkout to PR-head overwrites such overlays cleanly.
- **`install_steps.sh`** (auto-discovered at `tests/<recipe>/install_steps.sh`) — runs after - **`install_steps.sh`** (auto-discovered at `tests/<recipe>/install_steps.sh`) — runs after
`abra app new` + EXTRA_ENV + secret-generate, BEFORE the single deploy, with `CCCI_APP_DOMAIN` / `abra app new` + EXTRA_ENV + secret-generate, BEFORE the single deploy, with `CCCI_APP_DOMAIN` /
`CCCI_APP_ENV` / `CCCI_RECIPE` (and `CCCI_DEPS_FILE` when DEPS are provisioned at install). Use it to `CCCI_APP_ENV` / `CCCI_RECIPE` (and `CCCI_DEPS_FILE` when the recipe declares DEPS — deps are
drop a cc-ci-owned compose overlay into the checkout, wire dep-derived env/secrets, etc. always provisioned before the deploy). Use it to wire dep-derived env/secrets, seed config, etc.
**Non-HTTP protocol tests (mumble).** Reach a TCP service published `mode: host` (via a host-ports **Non-HTTP protocol tests (mumble).** Reach a TCP service published `mode: host` (via a host-ports
overlay) at `127.0.0.1:<port>` — cc-ci runs tests on-host (cc-ci-run). mumble ships a stdlib protocol overlay) at `127.0.0.1:<port>` — cc-ci runs tests on-host (cc-ci-run). mumble ships a stdlib protocol
@ -227,9 +235,10 @@ RECIPE=<recipe> PR=<n> REF=<sha-or-branch> SRC=recipe-maintainers/<recipe> \
``` ```
tests/lasuite-docs/ tests/lasuite-docs/
├── recipe_meta.py # HEALTH_PATH="/", DEPLOY_TIMEOUT=900, EXTRA_ENV(domain) for cold-pull, ├── recipe_meta.py # HEALTH_PATH="/", DEPLOY_TIMEOUT=900, EXTRA_ENV(ctx) for cold-pull,
│ # DEPS=["keycloak"] ← Phase 2 dep declaration │ # DEPS=["keycloak"] ← Phase 2 dep declaration
├── ops.py # pre_<op> seed hooks (volume marker for backup/restore data-integrity) ├── install_steps.sh # wires OIDC env from $CCCI_DEPS_FILE into the single deploy
├── ops.py # pre_<op>(ctx) seed hooks (volume marker for backup/restore data-integrity)
├── test_install.py # lifecycle install overlay (Playwright frontend SPA load) ├── test_install.py # lifecycle install overlay (Playwright frontend SPA load)
├── test_upgrade.py # lifecycle upgrade overlay (marker survives chaos redeploy) ├── test_upgrade.py # lifecycle upgrade overlay (marker survives chaos redeploy)
├── test_backup.py # lifecycle backup overlay (marker captured) ├── test_backup.py # lifecycle backup overlay (marker captured)
@ -239,12 +248,14 @@ tests/lasuite-docs/
├── test_health_check.py # parity port (SOURCE comment cites recipe-info file) ├── test_health_check.py # parity port (SOURCE comment cites recipe-info file)
├── test_auth_required.py # specific: /api/v1.0/users/me/ → 401 without auth ├── test_auth_required.py # specific: /api/v1.0/users/me/ → 401 without auth
└── test_oidc_with_keycloak.py # specific: full OIDC flow against the dep keycloak (uses └── test_oidc_with_keycloak.py # specific: full OIDC flow against the dep keycloak (uses
# harness.sso primitives + deps_apps["keycloak"]) # harness.sso primitives + the `deps` fixture)
``` ```
`!testme` on a lasuite-docs PR drives the orchestrator to: `!testme` on a lasuite-docs PR drives the orchestrator to:
1. Deploy the per-run keycloak dep (`keyc-<6hex>.ci.commoninternet.net`) and wait healthy. 1. Provision the per-run keycloak dep (`keyc-<6hex>.ci.commoninternet.net`), wait healthy, write
2. Deploy lasuite-docs (`lasu-<6hex>.ci.commoninternet.net`). creds to `$CCCI_DEPS_FILE` — BEFORE the recipe deploy.
2. Deploy lasuite-docs (`lasu-<6hex>.ci.commoninternet.net`); `install_steps.sh` wires the OIDC
env into that one deploy.
3. Run install / upgrade / backup / restore + the 3 functional tests against the shared 3. Run install / upgrade / backup / restore + the 3 functional tests against the shared
deployment (custom tier). deployment (custom tier).
4. Teardown lasuite-docs, then the keycloak dep (LAST), both with verify=True. 4. Teardown lasuite-docs, then the keycloak dep (LAST), both with verify=True.
@ -254,12 +265,13 @@ tests/lasuite-docs/
### Other shapes (concrete references) ### Other shapes (concrete references)
- **TCP / voice recipe — `tests/mumble/`**: `recipe_meta.py` (EXTRA_ENV sets - **TCP / voice recipe — `tests/mumble/`**: `recipe_meta.py` (EXTRA_ENV sets
`COMPOSE_FILE=compose.yml:compose.mumbleweb.yml:compose.host-ports.yml`, `WELCOME_TEXT`/`USERS` `COMPOSE_FILE=compose.yml:compose.mumbleweb.yml` for the base; `UPGRADE_EXTRA_ENV` adds the
markers, `CHAOS_BASE_DEPLOY=True`, `READY_PROBE` TCP 64738), `install_steps.sh` (provides the native `compose.host-ports.yml` at PR-head so 64738 is host-published on latest; private
host-ports overlay to older versions), `functional/_mumble_proto.py` + the protocol/config-round-trip `_WELCOME_TEXT_MARKER`/`_MAX_USERS` constants; `READY_PROBE(ctx)` TCP 64738 — phase-aware via
the live COMPOSE_FILE), `functional/_mumble_proto.py` + the protocol/config-round-trip
tests, `ops.py`/`test_backup.py`/`test_restore.py` (sqlite P4). See §2.4. tests, `ops.py`/`test_backup.py`/`test_restore.py` (sqlite P4). See §2.4.
- **Multi-service, dep-less, in-container functional — `tests/mailu/`**: `recipe_meta.py` - **Multi-service, dep-less, in-container functional — `tests/mailu/`**: `recipe_meta.py`
(`EXTRA_ENV(domain)` with `TLS_FLAVOR=notls` + `MAIL_DOMAIN`/`HOSTNAMES`/`TRAEFIK_STACK_NAME`), (`EXTRA_ENV(ctx)` with `TLS_FLAVOR=notls` + `MAIL_DOMAIN`/`HOSTNAMES`/`TRAEFIK_STACK_NAME`),
`functional/_mailu.py` (flask-CLI helpers), `test_mailbox.py` (create→config-export read-back), `functional/_mailu.py` (flask-CLI helpers), `test_mailbox.py` (create→config-export read-back),
`test_mail_flow.py` (in-container sendmail→doveadm delivery). No backupbot → P4 N/A (PARITY.md + `test_mail_flow.py` (in-container sendmail→doveadm delivery). No backupbot → P4 N/A (PARITY.md +
DEFERRED.md). See §2.4. DEFERRED.md). See §2.4.

385
docs/recipe-customization.md
View File

@ -1,8 +1,9 @@
# Recipe customization — review spec # Recipe customization — reference
Status: REVIEW SPEC — describes the customization surface as it exists today (main), written so Status: REFERENCE — describes the customization system as restructured on branch
the structure can be reviewed and potentially restructured. §8 lists known limitations and `restructure/recipe-custom` (the "rcust" restructure). The pre-restructure system and its defects
restructuring candidates; everything before it is purely descriptive. are documented in this file's history (commit `76a4b6b`, the review spec whose §8 R1–R9 drove the
restructure); §8 below records how each was resolved.
Companion docs: `docs/testing.md` (test architecture / tier semantics), `docs/enroll-recipe.md` Companion docs: `docs/testing.md` (test architecture / tier semantics), `docs/enroll-recipe.md`
(step-by-step enrollment). This doc is the **complete reference** for the two questions those docs (step-by-step enrollment). This doc is the **complete reference** for the two questions those docs
@ -15,17 +16,18 @@ answer only partially:
## 1. The three customization surfaces ## 1. The three customization surfaces
A recipe customizes its CI through **three distinct mechanisms** (worth noticing for the A recipe customizes its CI through **three distinct mechanisms**:
restructure review — they are three different config languages):
| Surface | Form | Examples | | Surface | Form | Examples |
|---|---|---| |---|---|---|
| **Declarative settings** | Python assignments in `tests/<recipe>/recipe_meta.py` | `DEPLOY_TIMEOUT = 1500`, `UPGRADE_BASE_VERSION = "2.3.1+..."` | | **Declarative settings** | Python assignments in `tests/<recipe>/recipe_meta.py` | `DEPLOY_TIMEOUT = 1500`, `UPGRADE_BASE_VERSION = "2.3.1+..."` |
| **Code hooks** | Callables in `recipe_meta.py`, `ops.py` functions, shell hooks | `def READY_PROBE(domain): ...`, `pre_upgrade()`, `install_steps.sh` | | **Code hooks** | Callables in `recipe_meta.py`, `ops.py` functions, one shell hook | `def READY_PROBE(ctx): ...`, `pre_upgrade(ctx)`, `install_steps.sh` |
| **File presence** | A file existing at a discovered path changes behavior | `test_upgrade.py` overlay, `functional/test_*.py`, `compose.ccci.yml` | | **File presence** | A file existing at a discovered path changes behavior | `test_upgrade.py` overlay, `functional/test_*.py`, `compose.ccci.yml` |
There is additionally a fourth, operator-facing surface: **environment variables** There is additionally a fourth, **operator-facing, local-dev-only** surface: environment variables
(`CCCI_SKIP_GENERIC*`) that override declarative settings at run time (§4.4). (`CCCI_SKIP_GENERIC*`) that suppress the generic floor at run time (§7). Whatever a run resolves
from all four surfaces is printed at run start as the **customization manifest** and embedded in
`results.json` under `"customization"` (§7) — one block answers "what does this recipe customize?".
## 2. Zero-config baseline ## 2. Zero-config baseline
@ -55,105 +57,98 @@ Two locations, with precedence and a security gate between them:
``` ```
tests/<recipe>/ # cc-ci side (repo-local mirrors the same shape) tests/<recipe>/ # cc-ci side (repo-local mirrors the same shape)
├── recipe_meta.py # ALL declarative settings + meta callables (§4) ├── recipe_meta.py # THE config file: registry-validated keys + ctx-hooks (§4)
├── test_<op>.py # lifecycle overlay assertions, op ∈ install|upgrade|backup|restore (§5.1) ├── test_<op>.py # lifecycle overlay assertions, op ∈ install|upgrade|backup|restore (§5.1)
├── ops.py # pre_<op>(domain, meta) seed hooks (§5.2) ├── ops.py # pre_<op>(ctx) seed hooks (§5.2)
├── test_*.py # custom-tier tests (top-level, cross-cutting)(§5.3)
├── functional/test_*.py # custom tier: parity ports + recipe-specific (§5.3) ├── functional/test_*.py # custom tier: parity ports + recipe-specific (§5.3)
├── playwright/test_*.py # custom tier: UI flows (§5.3) ├── playwright/test_*.py # custom tier: UI flows (§5.3)
├── install_steps.sh # pre-deploy shell hook (§5.4) ├── install_steps.sh # pre-deploy shell hook (the ONLY shell hook) (§5.4)
├── setup_custom_tests.sh # deps/OIDC credential wiring hook (§5.5) ├── compose.ccci.yml # CI-only compose overlay (first-class) (§5.5)
├── compose.ccci.yml # CI-only compose overlay (via install_steps) (§5.6)
└── PARITY.md # enrollment contract doc (human-read only) └── PARITY.md # enrollment contract doc (human-read only)
``` ```
**Placement rule (custom tests):** ALL custom-tier tests live under `functional/` or
`playwright/`. A top-level `test_*.py` is a lifecycle overlay (`test_<op>.py`) and nothing else —
top-level non-lifecycle files are NOT discovered (`discovery.custom_tests`; the lifecycle-name
exclusion stays as a safety net so a misfiled `test_<op>.py` can never double-run).
Precedence (machine-docs/DECISIONS.md, implemented in `discovery.py`): Precedence (machine-docs/DECISIONS.md, implemented in `discovery.py`):
- lifecycle overlay `test_<op>.py`: repo-local **wins** over cc-ci (same-name collision); the - lifecycle overlay `test_<op>.py`: repo-local **wins** over cc-ci (same-name collision); the
generic floor still runs additively alongside. generic floor still runs additively alongside.
- custom tier `test_*.py`: **ALL** run, from both locations (no collision concept). - custom tier (`functional/` + `playwright/`): **ALL** run, from both locations (no collision
concept).
- `install_steps.sh`: repo-local > cc-ci, or none. - `install_steps.sh`: repo-local > cc-ci, or none.
- `ops.py` pre-op hook: cc-ci wins; repo-local consulted only if approved. - `ops.py` pre-op hook: cc-ci wins; repo-local consulted only if approved.
- `recipe_meta.py`: cc-ci only — repo-local recipes cannot set CI settings (by design; the - `recipe_meta.py` and `compose.ccci.yml`: cc-ci only — repo-local recipes cannot set CI settings
settings surface stays maintainer-controlled). or compose overlays (by design; those surfaces stay maintainer-controlled).
## 4. `recipe_meta.py` — complete settings reference ## 4. `recipe_meta.py` — complete settings reference
The single settings file. Plain Python, `exec()`d by the harness (trusted, in-repo). A key is "set" The single settings file. Plain Python, `exec()`d by the harness in exactly ONE place: the
by a top-level assignment or `def`. Unknown names are ignored silently (a recipe may keep private registry-backed loader `runner/harness/meta.py::load(recipe) -> RecipeMeta`. Every consumer — the
constants here, e.g. mumble's `WELCOME_TEXT_MARKER` — but see §8 R6: typos in real key names are orchestrator (which loads once and passes the object down), the pytest `meta` fixture, lifecycle,
also silently ignored). deps, canonical, screenshot — reads from that one loaded object.
**Loader column legend** — this is the structural finding for the review (§8 R1). There is no **Validation (hard errors at load, before any deploy):**
single loader; six independent code paths each `exec()` the file and pick out their own keys:
| # | Loader | Keys it sees | - A key is "set" by a top-level ALL-CAPS assignment or `def`. Unknown ALL-CAPS top-level names
|---|---|---| raise `MetaError` listing the unknown name and the nearest registered key (typo gate —
| L1 | `runner/run_recipe_ci.py:_load_meta` (orchestrator) | 4 base + explicit 8-key allowlist | misspelling `READY_PROBE` can no longer silently disable the probe).
| L2 | `tests/conftest.py:_recipe_meta` (pytest `meta` fixture) | 4 base keys ONLY | - Type mismatches raise `MetaError`; callables are accepted only for hook-typed keys.
| L3 | `runner/harness/lifecycle.py:_recipe_extra_env` | `EXTRA_ENV` only | - **Underscore-prefixed names (`_FOO`) are recipe-private and exempt** — that's where private
| L4 | `runner/harness/lifecycle.py:_recipe_meta_flag` | boolean flags by name (`CHAOS_BASE_DEPLOY`) | constants live (e.g. mumble's `_WELCOME_TEXT_MARKER`). Lowercase names (helpers/imports) are
| L5 | `runner/harness/deps.py:declared_deps` | `DEPS` only | ignored.
| L6 | `runner/harness/canonical.py:is_canonical_enrolled` | `WARM_CANONICAL` only | - Hook callables must have the registered signature (below); a legacy-signature hook raises a
`MetaError` naming the migration, never a silent `TypeError` mid-run.
### 4.1 HTTP / health / timing (base 4 — seen by L1 AND L2) A unit test (`tests/unit/test_meta.py`) loads every `tests/*/recipe_meta.py` through the registry,
so a typo'd key fails at PR time, not at run time.
| Key | Type / default | Meaning | Used by | <!-- META-TABLE-START -->
_This table is GENERATED from the `runner/harness/meta.py` KEYS registry by `scripts/gen-meta-docs.py` — do not edit by hand (a unit test pins the sync)._
| Key | Type | Default | Meaning |
|---|---|---|---| |---|---|---|---|
| `HEALTH_PATH` | str, `"/"` | Path probed for serving/health checks | deploy wait (`lifecycle.py`), generic `assert_serving` | | `HEALTH_PATH` | `str` | `'/'` | Path probed for serving/health checks (deploy wait + generic `assert_serving`). |
| `HEALTH_OK` | tuple, `(200, 301, 302)` | Acceptable HTTP status codes for health | same | | `HEALTH_OK` | `tuple[int]` | `(200, 301, 302)` | Acceptable HTTP status codes for health. |
| `DEPLOY_TIMEOUT` | int s, `600` | Max wait for swarm convergence per deploy | `lifecycle.py`, generic ops | | `DEPLOY_TIMEOUT` | `int` | `600` | Max seconds to wait for swarm convergence per deploy. |
| `HTTP_TIMEOUT` | int s, `300` | Max wait for HTTP health after converged | same | | `HTTP_TIMEOUT` | `int` | `300` | Max seconds to wait for HTTP health after convergence. |
| `BACKUP_CAPABLE` | `bool` | `None` | Override the backup-tier capability auto-detect (compose `backupbot.backup` labels). `False` forces N/A; `True` forces the tier on; unset = auto-detect. |
| `EXPECTED_NA` | `dict` | `None` | Declare an N/A rung intentional: `{rung: reason}`. The cap stands either way; only the report wording changes. |
| `READY_PROBE` | `hook` | `None` | Callable `(ctx) -> [probe, ...]` returning extra readiness probes, run after install AND after upgrade: HTTP `{host, path, ok}` or TCP `{tcp_host, tcp_port, stable}`. |
| `UPGRADE_BASE_VERSION` | `str` | `None` | Exact published tag overriding the upgrade tier's base (default: `recipe_versions[-2]`). |
| `BACKUP_VERIFY` | `hook` | `None` | Callable `(ctx) -> bool` post-backup data-capture check; `False` re-runs the backup (truncated-dump race guard), retried up to 3 attempts. |
| `UPGRADE_EXTRA_ENV` | `dict_or_hook` | `None` | Extra `.env` keys applied after the PR-head checkout, before the chaos redeploy (env that exists only at head). Dict, or callable `(ctx) -> dict`. |
| `EXTRA_ENV` | `dict_or_hook` | `{}` | Extra `.env` keys applied at EVERY deploy (base install AND upgrade old-app). Dict, or callable `(ctx) -> dict` deriving values from the per-run domain (`ctx.domain`). |
| `DEPS` | `list[str]` | `[]` | Dep recipes deployed/provisioned alongside (e.g. `["keycloak"]`); creds land in `$CCCI_DEPS_FILE`. |
| `WARM_CANONICAL` | `bool` | `False` | Enroll the recipe in the warm/canonical app system (docs/warm.md): green cold runs on LATEST advance the canonical snapshot. |
| `SCREENSHOT` | `hook` | `None` | Callable `(page, ctx)` driving Playwright to a safe, credential-free post-login view for the results-card screenshot (default: landing page). |
Example: immich sets `DEPLOY_TIMEOUT = 1500`, `HTTP_TIMEOUT = 600` (ML containers are slow). <!-- META-TABLE-END -->
### 4.2 Upgrade tier (loader L1) ### 4.1 The uniform hook convention — `HookCtx`
| Key | Type / default | Meaning | Every recipe callable takes a single `ctx` argument (`harness/meta.py::HookCtx`, frozen):
|---|---|---|
| `UPGRADE_BASE_VERSION` | str (exact published tag), default `None` | **The "base pin"** — overrides the harness default base for the upgrade tier. Default base = `recipe_versions[-2]` (the previous published version); pin when that is not the PR's true predecessor (e.g. the PR is the first release on a new major, or the previous tag is known-broken). Must be an exact published tag — typos fail the base deploy. Consumed at `run_recipe_ci.py` (`prev = meta.get("UPGRADE_BASE_VERSION") or lifecycle.previous_version(recipe)`). Users: discourse, plausible. |
| `UPGRADE_EXTRA_ENV` | dict **or** callable `(domain) -> dict`, default `None` | Extra `.env` keys applied **after** the PR-head checkout, **before** the chaos redeploy (F2-14c) — for env vars that exist only at head (a new required setting introduced by the PR). Consumed in `generic.py:256`. User: mumble. |
### 4.3 Every-deploy shaping (loaders L3/L4 — NOT in the L1 allowlist) | Field | Meaning |
|---|---|
| `ctx.domain` | the app's per-run domain |
| `ctx.base_url` | `https://<domain>` |
| `ctx.meta` | the recipe's full `RecipeMeta` |
| `ctx.deps` | provisioned dep creds (`{dep_recipe: entry}`) or `None` |
| `ctx.op` | current lifecycle op (`install`/`upgrade`/`backup`/`restore`) or `None` |
| Key | Type / default | Meaning | Signatures: `EXTRA_ENV(ctx)`, `UPGRADE_EXTRA_ENV(ctx)`, `READY_PROBE(ctx)`, `BACKUP_VERIFY(ctx)`,
|---|---|---| `SCREENSHOT(page, ctx)`, ops.py `pre_<op>(ctx)`. Dict-valued `EXTRA_ENV`/`UPGRADE_EXTRA_ENV`
| `EXTRA_ENV` | dict **or** callable `(domain) -> dict`, default `{}` | Extra `.env` keys applied at **every** deploy (base install AND upgrade old-app). Callable form derives values from the per-run domain (e.g. cryptpad's `SANDBOX_DOMAIN`). Loaded by `lifecycle.py:_recipe_extra_env` (its own `exec()`). Users: cryptpad, discourse, ghost, matrix-synapse, mattermost-lts, mumble, plausible. | (non-callable) are still fine — only the callable form takes ctx. The loader enforces the
| `CHAOS_BASE_DEPLOY` | bool, default `False` | Base deploy uses `--chaos` so it survives untracked files in the recipe checkout (required when `install_steps.sh` copies in a `compose.ccci.yml` overlay — §5.6; implicit coupling, see §8 R7). Loaded by `lifecycle.py:_recipe_meta_flag`. Users: discourse, ghost. | parameter names at load time (a pre-restructure `(domain)`/`(domain, meta)` hook gets a pointed
`MetaError`, not a mid-run crash).
### 4.4 Skips and intentional N/A (loader L1) Worked hook examples: cryptpad (`EXTRA_ENV(ctx)` derives `SANDBOX_DOMAIN` from `ctx.domain`),
mumble (`READY_PROBE(ctx)` TCP voice-port probe, `UPGRADE_EXTRA_ENV(ctx)` adds a head-only compose
| Key | Type / default | Meaning | overlay), ghost/discourse (`BACKUP_VERIFY(ctx)` dump-capture check).
|---|---|---|
| `SKIP_GENERIC` | list of op names or `"all"`/`"*"`, default `[]` | Suppress the generic floor for the listed ops (overlay becomes override instead of additive). Two env equivalents at run time: `CCCI_SKIP_GENERIC=1` (all ops), `CCCI_SKIP_GENERIC_<OP>=1` (one op). Currently set by **no enrolled recipe** (env form is the one used, ad hoc). |
| `EXPECTED_NA` | dict `{rung: reason}`, default `None` | Declares an N/A rung **intentional** (e.g. `{"backup": "stateless, nothing to back up"}`). Undeclared N/A is reported as an *unintentional coverage gap*. Both cap the achievable level — declaring does not un-cap, it only changes the report wording (`results.py`). User: custom-html-tiny. |
| `BACKUP_CAPABLE` | bool, default auto-detect | Overrides the backup-tier capability detection (scan of recipe compose files for `backupbot.backup` labels, `generic.py:34`). `False` forces N/A; `True` forces the tier on. Users: custom-html-bkp-bad/rst-bad (harness self-test recipes). |
### 4.5 Readiness & data-verification hooks (loader L1, callable values)
| Key | Type / default | Meaning |
|---|---|---|
| `READY_PROBE` | callable `(domain) -> [probe, ...]`, default `None` | Extra readiness probes run after install AND after upgrade, before that tier's assertions. Probe dicts: HTTP `{host, path, ok}` or TCP `{tcp_host, tcp_port, stable}` (`stable`: must stay connectable across 3 checks — for UDP-adjacent voice ports etc.). Consumed at `lifecycle.py:516`. Users: lasuite-drive, mumble (TCP voice port). |
| `BACKUP_VERIFY` | callable `(domain) -> bool`, default `None` | Post-backup data-capture check, retried — guards the truncated-dump race (backup snapshot taken before the seeded marker row hit disk). Return `False` → retry the backup, then fail. Users: discourse, ghost. |
### 4.6 Dependencies / SSO (loaders L5 + L1)
| Key | Type / default | Meaning |
|---|---|---|
| `DEPS` | list of recipe names, default `[]` | Dep recipes deployed alongside (e.g. `["keycloak"]`). Dep domain is `<dep[:4]>-<6hex>`, hashed from (parent, pr, ref, dep) — collision-free per run. Creds land in `$CCCI_DEPS_FILE` (JSON); tests use the `deps_apps` fixture; teardown deps LAST. Deploy-count guard becomes `1 + len(DEPS)`. Loaded by `deps.py:declared_deps`. Users: lasuite-docs/-drive/-meet. |
| `OIDC_AT_INSTALL` | bool, default `False` | Provision deps **before** the single base deploy so `install_steps.sh` can wire OIDC env into that one deploy (reads `$CCCI_DEPS_FILE`). Default (legacy) is post-deploy provisioning + a `setup_custom_tests.sh` redeploy. Consumed at `run_recipe_ci.py:514`. Users: lasuite-drive, lasuite-meet. |
### 4.7 Warm-canonical enrollment (loader L6)
| Key | Type / default | Meaning |
|---|---|---|
| `WARM_CANONICAL` | bool, default `False` | Enrolls the recipe in the warm/canonical app system (`docs/warm.md`): green COLD runs on LATEST advance the canonical snapshot; the nightly sweep iterates enrolled recipes. Loaded by `canonical.py:is_canonical_enrolled`. User: custom-html. |
### 4.8 Cosmetic (BROKEN — see §8 R2)
| Key | Type / default | Meaning |
|---|---|---|
| `SCREENSHOT` | callable `(page, domain, meta) -> None` | Drives Playwright to a safe post-login view for the results-card screenshot (default: landing page). **Currently unreachable from the CI path**: `screenshot.py:41` reads it from the meta dict the orchestrator passes (`run_recipe_ci.py:1056`), but the L1 allowlist never loads `SCREENSHOT`, so the hook is always `None`. No recipe sets it (consistent with it never having worked). |
## 5. Writing custom tests & hooks ## 5. Writing custom tests & hooks
@ -166,104 +161,122 @@ test runs additively against the same state.
Conventions (see `tests/immich/test_backup.py` etc.): Conventions (see `tests/immich/test_backup.py` etc.):
- use the `live_app` fixture (asserts `CCCI_APP_DOMAIN` is set, yields the domain) - use the `live_app` fixture (asserts `CCCI_APP_DOMAIN` is set, yields the domain)
- use the `meta` fixture for HEALTH_*/timeouts (note: only the 4 base keys — §8 R3) - use the `meta` fixture — the recipe's FULL validated `RecipeMeta` (attribute access)
- read op context from `$CCCI_OP_STATE_FILE` (JSON written by the orchestrator after the op: - use the `op_state` fixture for op context (versions, `snapshot_id`, artifact paths — the
versions, artifact paths) orchestrator's run-scoped op record; skips with a clear reason outside an orchestrator run)
- execute in-container checks via `harness.lifecycle.exec_in_app(domain, service, cmd)` - execute in-container checks via `harness.lifecycle.exec_in_app(domain, service, cmd)`
### 5.2 Pre-op seed hooks — `ops.py` ### 5.2 Pre-op seed hooks — `ops.py`
`def pre_<op>(domain, meta)` callables, imported and called by the orchestrator **before** `def pre_<op>(ctx)` callables, imported and called by the orchestrator **before** performing the
performing the op. This is where data gets seeded so the post-op overlay can assert on it: op. This is where data gets seeded so the post-op overlay can assert on it:
```python ```python
# tests/immich/ops.py (pattern) # tests/immich/ops.py (pattern)
def pre_upgrade(domain, meta): _psql(domain, "INSERT ... 'upgrade-survives'") def pre_upgrade(ctx): _psql(ctx.domain, "INSERT ... 'upgrade-survives'")
def pre_backup(domain, meta): _psql(domain, "INSERT ... 'original'") def pre_backup(ctx): _psql(ctx.domain, "INSERT ... 'original'")
def pre_restore(domain, meta): _psql(domain, "DROP TABLE ci_marker") # damage, restore must undo def pre_restore(ctx): _psql(ctx.domain, "DROP TABLE ci_marker") # damage, restore must undo
``` ```
Seed → op → assert is the whole pattern: `pre_backup` writes a marker, the orchestrator backs up, Seed → op → assert is the whole pattern: `pre_backup` writes a marker, the orchestrator backs up,
`pre_restore` destroys it, the orchestrator restores, `test_restore.py` asserts the marker is back. `pre_restore` destroys it, the orchestrator restores, `test_restore.py` asserts the marker is back.
### 5.3 Custom tier — `functional/`, `playwright/`, top-level `test_*.py` ### 5.3 Custom tier — `functional/` and `playwright/` ONLY
All non-lifecycle `test_*.py` (discovery: `discovery.py:custom_tests`, recursive over the All custom-tier tests live under `tests/<recipe>/functional/` or `tests/<recipe>/playwright/`
top-level dir + `functional/` + `playwright/`; files named `test_<op>.py` excluded). Run in the (discovery: `discovery.custom_tests`; the placement rule, §3). Run in the CUSTOM tier, after
CUSTOM tier, after restore, against the post-upgrade (PR-head) app. ALL discovered files run — restore, against the post-upgrade (PR-head) app. ALL discovered files run — cc-ci's and (if
cc-ci's and (if HC2-approved) repo-local's, additively. HC2-approved) repo-local's, additively.
Enrollment contract (`docs/enroll-recipe.md`): ≥2 NEW functional tests beyond ports of existing Enrollment contract (`docs/enroll-recipe.md`): ≥2 NEW functional tests beyond ports of existing
upstream checks; ported tests carry `SOURCE:` comments. Playwright tests get the shared upstream checks; ported tests carry `SOURCE:` comments. Playwright tests get the shared
browser/harness helpers (`harness.browser`); SSO recipes get `harness.sso` browser/harness helpers (`harness.browser`); SSO recipes get `harness.sso`
(`setup_keycloak_realm` — idempotent, `oidc_password_grant` — provider-pluggable). (`setup_keycloak_realm` — idempotent, `oidc_password_grant` — provider-pluggable). The documented
import toolbox for custom tests is `from harness import lifecycle, sso, browser`.
Tests gate on deps via `CCCI_DEPS_READY` (skip-with-reason when `0`; the skip is counted and Tests needing deps use the `deps` fixture (entries expose `.domain` plus the full creds dict) and
fails the run if deps were declared but unprovisionable — `run_recipe_ci.py:816`). carry `@pytest.mark.requires_deps` — when dep provisioning failed they skip with reason
`deps-not-ready` and the skip count is reported and FAILS a declared-deps run (F2-11; a green exit
must not mask an unrun SSO test). Fixtures replace direct `os.environ` reads — after the
restructure no recipe test parses env by hand.
### 5.4 Pre-deploy shell hook — `install_steps.sh` ### 5.4 Pre-deploy shell hook — `install_steps.sh`
Runs after `abra app new` + `EXTRA_ENV` application + secret generation, **before** the base The ONLY shell hook. Runs after `abra app new` + `EXTRA_ENV` application + secret generation,
deploy. For setup that must precede the first deploy: writing extra config files into the recipe **before** the single base deploy. For setup that must precede the first deploy: writing extra
checkout, copying in a `compose.ccci.yml` overlay (§5.6), editing `.env` beyond simple key=val. config files into the recipe checkout, editing `.env` beyond simple key=val, and — for recipes
with `DEPS` — wiring dep-derived OIDC env into the deploy (deps are always provisioned BEFORE the
deploy; install-time wiring is the only mode, so there is exactly one deploy and no post-deploy
redeploy hook).
Env contract: `CCCI_APP_DOMAIN`, `CCCI_RECIPE`, `CCCI_APP_ENV` (path to the app's `.env`), and — Env contract: `CCCI_APP_DOMAIN`, `CCCI_RECIPE`, `CCCI_APP_ENV` (path to the app's `.env`), and —
when `OIDC_AT_INSTALL` deps exist — `CCCI_DEPS_FILE`. Must locate the recipe checkout when `DEPS` is declared — `CCCI_DEPS_FILE` (jq-readable JSON of dep creds/URLs; see
ABRA_DIR-aware: `RECIPE_DIR="${ABRA_DIR:-${HOME}/.abra}/recipes/${CCCI_RECIPE}"` (per-run lasuite-drive/-meet/-docs for the pattern). Must locate the recipe checkout ABRA_DIR-aware:
`ABRA_DIR` since the concurrency restructure — a hardcoded `~/.abra` writes to the wrong tree). `RECIPE_DIR="${ABRA_DIR:-${HOME}/.abra}/recipes/${CCCI_RECIPE}"` (per-run `ABRA_DIR` since the
concurrency restructure — a hardcoded `~/.abra` writes to the wrong tree).
Graceful-generic rule: a recipe needing a hook but not shipping one simply fails the generic Graceful-generic rule: a recipe needing a hook but not shipping one simply fails the generic
install — a correct reported outcome, not a harness error. install — a correct reported outcome, not a harness error.
### 5.5 Deps credential wiring — `setup_custom_tests.sh` ### 5.5 CI-only compose overlay — `compose.ccci.yml`
For legacy (post-deploy) deps provisioning: runs after deps are up, reads `$CCCI_DEPS_FILE` **First-class:** if `tests/<recipe>/compose.ccci.yml` exists, the harness itself copies it into
(jq-readable JSON of dep creds/URLs), wires OIDC config via `abra app config set` + secrets, and the recipe checkout (ABRA_DIR-aware) before the base deploy and automatically uses `--chaos` for
redeploys. With `OIDC_AT_INSTALL = True` this hook is unnecessary (wiring happens in that deploy (the untracked file would otherwise trip abra's clean-tree gate). No
`install_steps.sh` before the only deploy) — preferred for new enrollments (one deploy, no `install_steps.sh` copy boilerplate, no flag to remember (the old `CHAOS_BASE_DEPLOY` ⇄ overlay
deploy-count exception). coupling is gone). The overlay is cc-ci-owned only.
### 5.6 CI-only compose overlay — `compose.ccci.yml` Policy unchanged: overlays are a minimal, justified fallback (ghost's is a 15m `start_period`
grace — a literal, because abra validates `start_period` before env substitution). Reference the
overlay from `EXTRA_ENV`'s `COMPOSE_FILE` as usual. Users: ghost, discourse.
Not auto-discovered: `install_steps.sh` copies it into the recipe checkout, and the recipe must ### 5.6 Environment & fixture contract (what custom code can read)
set `CHAOS_BASE_DEPLOY = True` so the base deploy (`--chaos`) tolerates the untracked file.
Policy: minimal, justified fallback only (ghost's is a 15m `start_period` grace — a literal,
because abra validates `start_period` before env substitution). The overlay is cc-ci-owned even
though it rides in the recipe checkout.
### 5.7 Environment contract summary (what custom code can read) Pytest fixtures (`tests/conftest.py` — the single fixture file):
| Fixture | Yields |
|---|---|
| `recipe` | the recipe name (`$RECIPE`) |
| `meta` | the FULL validated `RecipeMeta` (single loader) |
| `live_app` | the shared deployment's domain (asserts it exists) |
| `op_state` | the orchestrator's op-context dict (skips cleanly outside a run) |
| `deps` | `{dep_recipe: entry}` — entries expose `.domain` + full SSO creds |
Environment (hooks/shell, and approved repo-local code):
| Var | Set for | Meaning | | Var | Set for | Meaning |
|---|---|---| |---|---|---|
| `CCCI_APP_DOMAIN` | all tests + hooks | the app's per-run domain | | `CCCI_APP_DOMAIN` | all tests + hooks | the app's per-run domain |
| `CCCI_BASE_URL` | approved repo-local code | `https://<domain>` | | `CCCI_BASE_URL` | approved repo-local code | `https://<domain>` |
| `CCCI_RECIPE`, `CCCI_APP_ENV` | `install_steps.sh` | recipe name, app `.env` path | | `CCCI_RECIPE`, `CCCI_APP_ENV` | `install_steps.sh` | recipe name, app `.env` path |
| `CCCI_OP_STATE_FILE` | overlay tests | JSON op context (versions, artifacts) | | `CCCI_OP_STATE_FILE` | overlay tests (via `op_state`) | JSON op context (versions, artifacts) |
| `CCCI_DEPS_FILE` | deps hooks + tests | JSON dep creds dict | | `CCCI_DEPS_FILE` | `install_steps.sh` + harness | JSON dep creds dict |
| `CCCI_DEPS_READY` / `CCCI_DEPS_NOT_READY_REASON` | custom tier | gate SSO tests, skip-with-reason | | `CCCI_DEPS_READY` / `CCCI_DEPS_NOT_READY_REASON` | custom tier (via `requires_deps`) | gate SSO tests, skip-with-reason |
## 6. Run-model context (what the settings plug into) ## 6. Run-model context (what the settings plug into)
One deploy chain per run (full detail: `docs/testing.md` §2): One deploy chain per run (full detail: `docs/testing.md` §2):
``` ```
[DEPS? provision deps FIRST → $CCCI_DEPS_FILE]
deploy BASE (UPGRADE_BASE_VERSION or recipe_versions[-2]; EXTRA_ENV; install_steps.sh; deploy BASE (UPGRADE_BASE_VERSION or recipe_versions[-2]; EXTRA_ENV; install_steps.sh;
CHAOS_BASE_DEPLOY?; OIDC_AT_INSTALL deps first?) compose.ccci.yml auto-copied + auto-chaos)
→ INSTALL tier (READY_PROBE; generic + overlay asserts) → INSTALL tier (READY_PROBE; generic + overlay asserts)
→ pre_upgrade → chaos-deploy PR HEAD (UPGRADE_EXTRA_ENV) → pre_upgrade(ctx) → chaos-deploy PR HEAD (UPGRADE_EXTRA_ENV)
→ UPGRADE tier (READY_PROBE; version-label == head_ref) → UPGRADE tier (READY_PROBE; version-label == head_ref)
→ pre_backup → backup (BACKUP_CAPABLE; BACKUP_VERIFY) → pre_backup(ctx) → backup (BACKUP_CAPABLE; BACKUP_VERIFY)
→ BACKUP tier → BACKUP tier
→ pre_restore → restore → pre_restore(ctx) → restore
→ RESTORE tier → RESTORE tier
→ CUSTOM tier (functional/ + playwright/; deps via CCCI_DEPS_*) → CUSTOM tier (functional/ + playwright/; deps via the `deps` fixture)
→ SCREENSHOT (best-effort, never affects the verdict)
→ teardown (deps LAST) → teardown (deps LAST)
``` ```
Deploy-count guard (DG4.1): exactly `1 + len(DEPS)` deploys per run (chaos redeploys don't Deploy-count guard (DG4.1): exactly `1 + len(DEPS)` deploys per run (chaos redeploys don't
count); the per-run counter file is keyed by run since the concurrency restructure. count); the per-run counter file is keyed by run since the concurrency restructure.
## 7. Local iteration ## 7. Local iteration, the manifest, and the dev-only escape hatch
``` ```
RECIPE=<recipe> PR=<n> REF=<sha> SRC=recipe-maintainers/<recipe> \ RECIPE=<recipe> PR=<n> REF=<sha> SRC=recipe-maintainers/<recipe> \
@ -273,81 +286,75 @@ RECIPE=<recipe> PR=<n> REF=<sha> SRC=recipe-maintainers/<recipe> \
(`docs/enroll-recipe.md` §5 for the full loop, including dep teardown caveats.) (`docs/enroll-recipe.md` §5 for the full loop, including dep teardown caveats.)
## 8. Known limitations & restructuring candidates **Customization manifest.** Every run prints, right after meta load + discovery, one block:
The review section. Ordered by how much they'd shape a restructure. ```
===== customization manifest: <recipe> =====
meta (non-default): DEPLOY_TIMEOUT=1500 DEPS=['keycloak'] EXTRA_ENV='<hook>'
hooks: ops.py[pre_backup,pre_upgrade](cc-ci) install_steps.sh(cc-ci) compose.ccci.yml(cc-ci)
overlays: test_backup.py(cc-ci) test_restore.py(repo-local)
custom tests: functional/=5 playwright/=2 (cc-ci)
env overrides: (none)
```
**R1 — Six divergent meta loaders (the core drift hazard).** §4's L1–L6: every loader re-`exec()`s The same dict is embedded in `results.json` under `"customization"`. It is pure presentation —
`recipe_meta.py` and cherry-picks its own keys. Adding a key means knowing *which* loader to touch built from the SAME discovery/meta calls the run uses (so it cannot disagree with what executes,
(or that you must extend the L1 allowlist — `SCREENSHOT` proves people don't, R2). Two conventions and it honors the HC2 gate) — and never influences a verdict.
coexist: L1's explicit allowlist vs L3–L6's ad-hoc `ns.get(...)` which silently bypasses it.
*Candidate:* one `harness.meta.load(recipe) -> RecipeMeta` with a declarative key registry
(name, type, default, validator, consumer) as the single source of truth; L1–L6 become lookups
into the one loaded object; the registry also generates §4 of this doc (kills doc drift, R5).
**R2 — `SCREENSHOT` is a dead knob.** Fully implemented consumer (`screenshot.py`), documented **Dev-only generic skip.** `CCCI_SKIP_GENERIC=1` (all ops) / `CCCI_SKIP_GENERIC_<OP>=1` (one op)
hook contract, never reachable: the orchestrator's allowlist omits it, so the dict passed at suppress the generic floor — a LOCAL-DEV-ONLY escape hatch for iterating on one tier. There is no
`run_recipe_ci.py:1056` can never contain it. Direct evidence of R1. *Candidate:* fix trivially by declarative equivalent (the old `SKIP_GENERIC` meta key is deleted). If the env form is active in
adding to the allowlist — or delete the hook path if post-login screenshots aren't wanted; decide a CI (drone) run, the run prints a loud `!!` warning and the manifest records it.
during the restructure.
**R3 — The pytest `meta` fixture sees 4 keys.** `tests/conftest.py:_recipe_meta` loads only ## 8. Restructure outcomes (the review spec's R1–R9)
HEALTH_*/timeouts. An overlay test wanting e.g. `EXPECTED_NA` or a recipe constant must re-exec
the file itself. Probably intended minimalism, but it's a third key-set to keep in sync.
*Folds into R1.*
**R4 — Settings split across three config languages** (§1): recipe_meta keys, file-presence How each defect identified in the review spec (commit `76a4b6b` §8) was resolved:
(`install_steps.sh` existing changes deploy behavior), and run-time env (`CCCI_SKIP_GENERIC*`).
A reviewer asking "what does this recipe customize?" must check all three. *Candidate:* keep the
three surfaces (they serve different actors) but make the run header log a single resolved
"customization manifest" per run: every non-default key + every discovered hook file + every
CCCI_* override, in one block.
**R5 — Reference-doc drift already happened.** `docs/testing.md` documents 6 meta keys, - **R1 — six divergent meta loaders → RESOLVED.** One registry-backed loader
`docs/enroll-recipe.md` shows others by example; neither is complete (18 keys exist). This doc is (`harness/meta.py::load`), the only `exec()` of `recipe_meta.py`. The orchestrator loads once
now complete but handwritten — it will drift too. *Candidate:* generate the key table from the R1 and passes the `RecipeMeta` down; conftest/lifecycle/deps/canonical all read the one object.
registry (test asserts doc ⊆ registry). - **R2 — dead `SCREENSHOT` knob → RESOLVED (kept + fixed).** The registry replaced the allowlist
that orphaned it; the orchestrator path now delivers the hook to `screenshot.py`
(proven end-to-end by `tests/unit/test_screenshot.py::test_screenshot_reachable_through_real_load_path`).
- **R3 — 4-key pytest `meta` fixture → RESOLVED.** The fixture returns the full validated
`RecipeMeta`.
- **R4 — three config languages → MITIGATED by the manifest** (§7): the surfaces stay (they serve
different actors), but every run resolves them into one visible block + results key.
- **R5 — reference-doc drift → RESOLVED.** §4's key table is generated from the registry
(`scripts/gen-meta-docs.py`); a unit test fails CI on drift; `testing.md`/`enroll-recipe.md`
point here instead of keeping partial lists.
- **R6 — silent typos → RESOLVED.** Unknown ALL-CAPS keys and type mismatches are hard
`MetaError`s; private constants are underscore-prefixed (exempt).
- **R7 — `compose.ccci.yml` ⇄ `CHAOS_BASE_DEPLOY` coupling → RESOLVED.** The overlay is
first-class: harness-copied, auto-chaos. The flag is deleted.
- **R8 — zero-user `SKIP_GENERIC` meta key → RESOLVED (deleted).** Env form remains, documented
dev-only, loudly flagged in CI runs (§7).
- **R9 — `recipe_meta.py` is code, not config → REJECTED by decision.** No data/hooks file split:
registry validation gets the value (typed, validated keys) at lower cost; one file per recipe
remains the single config place. The expressiveness need is real (cryptpad derives env from the
per-run domain).
**R6 — No schema validation / silent typos.** Unknown top-level names in `recipe_meta.py` are Also settled in the restructure: install-time deps provisioning is the ONLY mode (the legacy
ignored, which is load-bearing (recipes keep private constants there: mumble's post-deploy `setup_custom_tests.sh` machinery and its extra redeploy are deleted); the custom-test
`WELCOME_TEXT_MARKER`, `MAX_USERS`). Consequence: misspelling `READY_PROBE` as `READINESS_PROBE` placement rule (§3); the uniform ctx hook convention (§4.1); the consolidated fixture surface
silently disables the probe — the run goes green with less coverage, the worst failure mode for a (§5.6 — `deps` replaces `deps_apps`+`deps_creds`; dead `deployed`/`deployed_app`/`app_domain`
CI harness. *Candidate:* with the R1 registry, warn (not fail) on ALL-CAPS top-level names that fixtures deleted).
are not registered and not referenced by the recipe's own tests; or namespace private constants
(`_WELCOME_TEXT_MARKER`).
**R7 — `compose.ccci.yml` ⇄ `CHAOS_BASE_DEPLOY` implicit coupling.** The overlay only works if
the recipe *also* sets the flag; forgetting it fails the base deploy with an abra
untracked-files error far from the cause. *Candidate:* if `install_steps.sh` exists alongside a
`compose.ccci.yml`, the harness could auto-enable chaos for the base deploy (or at least assert
the flag and fail with a pointed message).
**R8 — `SKIP_GENERIC` (meta form) has zero users.** Only the env-var form is used, ad hoc. Either
the meta key earns its place (first real user) or it's surface to delete in the restructure.
**R9 — `recipe_meta.py` is code, not config.** Five keys take callables (`EXTRA_ENV`,
`UPGRADE_EXTRA_ENV`, `READY_PROBE`, `BACKUP_VERIFY`, `SCREENSHOT`), so the file must stay an
`exec()`d Python module — it can't be validated as data, serialized into results, or diffed
declaratively. This is a real expressiveness need (cryptpad derives `SANDBOX_DOMAIN` from the
per-run domain), not an accident. *Candidate if restructuring:* split data keys (TOML-able,
schema-validated) from a `hooks.py` (callables only) — but weigh against the cost of two files
per recipe; the R1 registry gets most of the value without the split.
## 9. File / symbol index ## 9. File / symbol index
| Concern | Where | | Concern | Where |
|---|---| |---|---|
| Orchestrator meta loader (L1, allowlist) | `runner/run_recipe_ci.py:250` `_load_meta` | | THE meta loader + key registry + `HookCtx` + `MetaError` | `runner/harness/meta.py` (`load`, `KEYS`, `check_hook_signature`) |
| Pytest meta fixture (L2) | `tests/conftest.py` `_recipe_meta` | | Generated key table | `scripts/gen-meta-docs.py` → §4 above (sync pinned by `tests/unit/test_meta.py`) |
| `EXTRA_ENV` loader (L3) | `runner/harness/lifecycle.py:114` `_recipe_extra_env` | | Customization manifest | `runner/harness/manifest.py` (`build`, `render`), printed by `runner/run_recipe_ci.py` |
| Boolean-flag loader (L4) | `runner/harness/lifecycle.py:132` `_recipe_meta_flag` | | Overlay/custom/hook discovery + HC2 gate + placement rule | `runner/harness/discovery.py` |
| `DEPS` loader (L5) | `runner/harness/deps.py:37` `declared_deps` |
| `WARM_CANONICAL` loader (L6) | `runner/harness/canonical.py:36` `is_canonical_enrolled` |
| Overlay/custom/hook discovery + HC2 gate | `runner/harness/discovery.py` |
| HC2 allowlist | `tests/repo-local-approved.txt` | | HC2 allowlist | `tests/repo-local-approved.txt` |
| Generic assertions + `BACKUP_CAPABLE` detect | `runner/harness/generic.py` | | Generic assertions + `BACKUP_CAPABLE` detect | `runner/harness/generic.py` |
| `READY_PROBE` / `CHAOS_BASE_DEPLOY` consumption | `runner/harness/lifecycle.py:516` / `:283` | | `compose.ccci.yml` auto-copy + auto-chaos | `runner/harness/lifecycle.py` (`provide_ccci_overlay`, `deploy_app`) |
| `READY_PROBE` consumption | `runner/harness/lifecycle.py` (`wait_ready_probes`) |
| `EXPECTED_NA` reporting | `runner/harness/results.py` | | `EXPECTED_NA` reporting | `runner/harness/results.py` |
| Dead `SCREENSHOT` consumer | `runner/harness/screenshot.py:36`, called `run_recipe_ci.py:1056` | | `SCREENSHOT` consumer | `runner/harness/screenshot.py` |
| Skip-generic logic (meta + env) | `runner/run_recipe_ci.py:285` | | Fixtures (`recipe`/`meta`/`live_app`/`op_state`/`deps`) + F2-11 skip-report | `tests/conftest.py` |
| Worked examples | `tests/ghost/` (overlay+chaos), `tests/mumble/` (TCP probe, UPGRADE_EXTRA_ENV), `tests/lasuite-drive/` (DEPS+OIDC_AT_INSTALL), `tests/immich/` (ops.py seed pattern) | | Skip-generic env logic (dev-only) | `runner/run_recipe_ci.py` (`_skip_generic`) |
| Unit tests pinning all of the above | `tests/unit/test_meta.py`, `test_manifest.py`, `test_discovery*.py` |
| Worked examples | `tests/ghost/` (overlay+compose.ccci.yml), `tests/mumble/` (TCP probe, UPGRADE_EXTRA_ENV, private `_` constants), `tests/lasuite-drive/` (DEPS + install-time OIDC wiring), `tests/immich/` (ops.py seed pattern) |

61
docs/testing.md
View File

@ -16,12 +16,13 @@ year from now, this is the one rule that should still hold.
ship as the floor for every recipe. No SSO provider, no external deps, no per-recipe state ship as the floor for every recipe. No SSO provider, no external deps, no per-recipe state
scaffolding — just "does this recipe deploy and lifecycle work?" scaffolding — just "does this recipe deploy and lifecycle work?"
- **Generic must not depend on custom.** A custom test or a custom-tests setup (e.g. SSO/OIDC dep - **Generic must not depend on custom.** A custom test or a custom-tests setup (e.g. SSO/OIDC dep
provisioning) **can never be a precondition for the generic tier to pass.** Concretely: the provisioning) **can never be a precondition for the generic tier to pass.** Concretely: deps are
orchestrator runs all generic tiers (install → upgrade → backup → restore) against the recipe provisioned BEFORE the single deploy (so `install_steps.sh` can wire OIDC env into that one
**alone, with no deps deployed**, then runs the `setup_custom_tests` step (deps + post-deps deploy), but a dep-provisioning failure is **isolated** to the custom tier — the recipe still
wiring) only after — and a failure there is **isolated** to the custom tier (tests tagged deploys alone, every generic tier (install → upgrade → backup → restore) runs normally, and
`@pytest.mark.requires_deps` skip with reason `"deps-not-ready"`; generic tier reports tests tagged `@pytest.mark.requires_deps` skip with reason `"deps-not-ready"` (a counted,
normally). See `cc-ci-plan/plan-sso-dep-testing.md` for the SSO-dep specifics. reported skip — F2-11). A deps failure can never fail or block a generic tier. See
`cc-ci-plan/plan-sso-dep-testing.md` for the SSO-dep specifics.
- **Custom tests are the thoroughness layer — and they cost more to maintain.** They're more - **Custom tests are the thoroughness layer — and they cost more to maintain.** They're more
thorough (authenticated APIs, multi-app flows, version-specific browser selectors, helper thorough (authenticated APIs, multi-app flows, version-specific browser selectors, helper
scripts, state-management) and *therefore* take more maintenance: an SSO provider's admin API scripts, state-management) and *therefore* take more maintenance: an SSO provider's admin API
@ -113,9 +114,11 @@ repo-local <recipe-repo>/tests/test_<op>.py (upstream-authoritative; gated
Only ONE overlay source wins for a given op (repo-local > cc-ci); the generic floor runs **in Only ONE overlay source wins for a given op (repo-local > cc-ci); the generic floor runs **in
addition** unless explicitly opted out. addition** unless explicitly opted out.
**Custom (non-lifecycle) `test_*.py`** — any other `test_*.py` (e.g. `test_sso.py`) is **opt-in and **Custom (non-lifecycle) tests** — e.g. `functional/test_sso.py` — are **opt-in and additive**:
additive**: it has no generic equivalent and runs only when present, discovered from both locations they have no generic equivalent and run only when present, discovered from both locations
(repo-local gated by the HC2 allowlist). (repo-local gated by the HC2 allowlist). Placement rule: custom tests live ONLY under
`functional/` or `playwright/`; a top-level `test_*.py` is a lifecycle overlay and nothing else
(top-level non-lifecycle files are not discovered).
### Pre-op seed hooks (per-recipe `ops.py`) ### Pre-op seed hooks (per-recipe `ops.py`)
@ -127,35 +130,38 @@ etc.). Since the orchestrator owns the op, overlays place their seed in an optio
# tests/<recipe>/ops.py # tests/<recipe>/ops.py
from harness import lifecycle from harness import lifecycle
def pre_upgrade(domain, meta): def pre_upgrade(ctx):
# seed a marker before the harness performs the upgrade # seed a marker before the harness performs the upgrade
lifecycle.exec_in_app(domain, ["sh", "-c", "echo upgrade-survives > /path/marker"]) lifecycle.exec_in_app(ctx.domain, ["sh", "-c", "echo upgrade-survives > /path/marker"])
def pre_backup(domain, meta): def pre_backup(ctx):
# establish a known "original" state before the backup op captures it # establish a known "original" state before the backup op captures it
lifecycle.exec_in_app(domain, ["sh", "-c", "echo original > /path/marker"]) lifecycle.exec_in_app(ctx.domain, ["sh", "-c", "echo original > /path/marker"])
def pre_restore(domain, meta): def pre_restore(ctx):
# diverge from the backed-up state so a successful restore is observable # diverge from the backed-up state so a successful restore is observable
lifecycle.exec_in_app(domain, ["sh", "-c", "echo mutated > /path/marker"]) lifecycle.exec_in_app(ctx.domain, ["sh", "-c", "echo mutated > /path/marker"])
``` ```
The orchestrator imports `ops.py` in-process (with the recipe dir on `sys.path`, so it can import The orchestrator imports `ops.py` in-process (with the recipe dir on `sys.path`, so it can import
sibling helpers like `kc_admin.py`) and calls `pre_<op>(domain, meta)` immediately before performing sibling helpers like `kc_admin.py`) and calls `pre_<op>(ctx)` immediately before performing the
the op. Then `test_<op>.py` asserts the post-op state. See `tests/custom-html/` (volume marker), op — `ctx` is the uniform `HookCtx` every recipe hook receives (`.domain`, `.base_url`, `.meta`,
`.deps`, `.op` — `docs/recipe-customization.md` §4.1). Then `test_<op>.py` asserts the post-op
state. See `tests/custom-html/` (volume marker),
`tests/keycloak/` (admin-API/realm), `tests/matrix-synapse/`, `tests/lasuite-docs/` (psql in the `db` `tests/keycloak/` (admin-API/realm), `tests/matrix-synapse/`, `tests/lasuite-docs/` (psql in the `db`
service) for worked examples. service) for worked examples.
### Opting out of the generic floor ### Opting out of the generic floor (LOCAL-DEV-ONLY)
The generic runs additively by default. To skip it (e.g. when an overlay's recipe-specific check The generic runs additively by default and there is **no declarative opt-out** — no recipe can
fully replaces the generic's mechanism check) set, in increasing specificity: ship without the floor. For local iteration only (e.g. re-running one tier while developing an
overlay), two env escape hatches exist:
- **env `CCCI_SKIP_GENERIC=1`** — skip generic for ALL ops (run-wide). - **env `CCCI_SKIP_GENERIC=1`** — skip generic for ALL ops (run-wide).
- **env `CCCI_SKIP_GENERIC_<OP>=1`** — e.g. `CCCI_SKIP_GENERIC_UPGRADE=1` — skip generic for that one op. - **env `CCCI_SKIP_GENERIC_<OP>=1`** — e.g. `CCCI_SKIP_GENERIC_UPGRADE=1` — skip generic for that one op.
- **declarative in `recipe_meta.py`** — `SKIP_GENERIC = ["upgrade"]` (per-op) or `SKIP_GENERIC = ["all"]`.
Opting out is per-recipe and visible in git — not a hidden global. Truthy = `1`/`true`/`yes`/`on`. Truthy = `1`/`true`/`yes`/`on`. If either is active in a CI (drone) run, the run prints a loud
`!!` warning and the customization manifest records it (`docs/recipe-customization.md` §7).
## Repo-local trust gate (HC2) — default-deny ## Repo-local trust gate (HC2) — default-deny
@ -215,12 +221,14 @@ installs and stays 1.
`tests/custom-html/test_upgrade.py`). Assert the POST-op state — reading app state through `tests/custom-html/test_upgrade.py`). Assert the POST-op state — reading app state through
`lifecycle.exec_in_app` (volume/DB) for data checks, not HTTP. Generic + your overlay both run. `lifecycle.exec_in_app` (volume/DB) for data checks, not HTTP. Generic + your overlay both run.
3. If the overlay needs to seed PRE-op state (data-continuity markers, the backup→restore 3. If the overlay needs to seed PRE-op state (data-continuity markers, the backup→restore
divergence), drop `tests/<recipe>/ops.py` with `pre_upgrade/pre_backup/pre_restore(domain, meta)`. divergence), drop `tests/<recipe>/ops.py` with `pre_upgrade/pre_backup/pre_restore(ctx)`.
4. If the recipe needs install-time setup, add `tests/<recipe>/install_steps.sh`. 4. If the recipe needs install-time setup, add `tests/<recipe>/install_steps.sh`.
5. Set per-recipe knobs (health path, timeouts, opt-out) in `recipe_meta.py`. 5. Set per-recipe knobs (health path, timeouts) in `recipe_meta.py`.
6. **Never weaken or skip an assertion to make a run pass** — a red tier is information. 6. **Never weaken or skip an assertion to make a run pass** — a red tier is information.
Per-recipe config (`tests/<recipe>/recipe_meta.py`, all optional): Per-recipe config (`tests/<recipe>/recipe_meta.py`, all optional — the COMPLETE key reference is
the generated table in `docs/recipe-customization.md` §4; unknown keys are hard errors, private
constants are underscore-prefixed):
```python ```python
HEALTH_PATH = "/realms/master" # path that returns a healthy status (default "/") HEALTH_PATH = "/realms/master" # path that returns a healthy status (default "/")
@ -228,8 +236,7 @@ HEALTH_OK = (200,) # acceptable status codes (default 200/301/302)
DEPLOY_TIMEOUT = 600 # seconds for services to converge (default 600) DEPLOY_TIMEOUT = 600 # seconds for services to converge (default 600)
HTTP_TIMEOUT = 600 # seconds for the app to answer (default 300) HTTP_TIMEOUT = 600 # seconds for the app to answer (default 300)
BACKUP_CAPABLE = True # override backup-capability auto-detection (default: scan compose) BACKUP_CAPABLE = True # override backup-capability auto-detection (default: scan compose)
EXTRA_ENV = {"KEY": "value"} # or EXTRA_ENV(domain) -> dict; extra .env keys set at deploy EXTRA_ENV = {"KEY": "value"} # or EXTRA_ENV(ctx) -> dict; extra .env keys set at deploy
SKIP_GENERIC = ["upgrade"] # per-recipe declarative opt-out from generic ops ("all" = every op)
``` ```
The harness self-tests for discovery / precedence / the HC2 allowlist live in `tests/unit/` (run: The harness self-tests for discovery / precedence / the HC2 allowlist live in `tests/unit/` (run:

14
runner/harness/canonical.py
View File

@ -30,17 +30,13 @@ import subprocess
import time import time
from . import abra, warm, warmsnap from . import abra, warm, warmsnap
from . import meta as meta_mod
def is_enrolled(recipe: str) -> bool: def is_enrolled(recipe: str) -> bool:
"""True if `tests/<recipe>/recipe_meta.py` sets `WARM_CANONICAL = True`. Missing meta → False.""" """True if `tests/<recipe>/recipe_meta.py` sets `WARM_CANONICAL = True`. Missing meta → False.
path = os.path.join(os.path.dirname(__file__), "..", "..", "tests", recipe, "recipe_meta.py") Reads through the single meta loader (rcust P1 — no per-module exec)."""
if not os.path.exists(path): return bool(meta_mod.load(recipe).WARM_CANONICAL)
return False
ns: dict = {}
with open(path) as fh:
exec(compile(fh.read(), path, "exec"), ns) # noqa: S102 (trusted, in-repo)
return bool(ns.get("WARM_CANONICAL"))
def canonical_domain(recipe: str) -> str: def canonical_domain(recipe: str) -> str:
@ -51,7 +47,7 @@ def canonical_domain(recipe: str) -> str:
def enrolled_recipes() -> list[str]: def enrolled_recipes() -> list[str]:
"""All recipes enrolled as data-warm canonicals (recipe_meta.WARM_CANONICAL=True), sorted. Used """All recipes enrolled as data-warm canonicals (recipe_meta.WARM_CANONICAL=True), sorted. Used
by the WC6 nightly sweep to know which canonicals to refresh via a green cold run on latest.""" by the WC6 nightly sweep to know which canonicals to refresh via a green cold run on latest."""
tests_dir = os.path.join(os.path.dirname(__file__), "..", "..", "tests") tests_dir = meta_mod.TESTS_DIR
out = [] out = []
try: try:
for name in sorted(os.listdir(tests_dir)): for name in sorted(os.listdir(tests_dir)):

42
runner/harness/deps.py
View File

@ -20,7 +20,7 @@ Per Phase-2 DECISIONS:
Run state: Run state:
- `$CCCI_DEPS_FILE` — JSON file written by the orchestrator after each dep deploys; each entry is - `$CCCI_DEPS_FILE` — JSON file written by the orchestrator after each dep deploys; each entry is
`{"recipe": "<dep-recipe>", "domain": "<dep-domain>", "version": null}`. Tests access via the `{"recipe": "<dep-recipe>", "domain": "<dep-domain>", "version": null}`. Tests access via the
`deps_apps` pytest fixture defined in `tests/conftest.py`. `deps` pytest fixture defined in `tests/conftest.py`.
""" """
from __future__ import annotations from __future__ import annotations
@ -31,19 +31,7 @@ import os
from collections.abc import Iterable from collections.abc import Iterable
from . import lifecycle, naming from . import lifecycle, naming
from . import meta as meta_mod
def declared_deps(recipe: str) -> list[str]:
"""Read `DEPS` from `tests/<recipe>/recipe_meta.py` — a list of recipe names this recipe needs
deployed alongside it. Returns [] if none."""
path = os.path.join(os.path.dirname(__file__), "..", "..", "tests", recipe, "recipe_meta.py")
if not os.path.exists(path):
return []
ns: dict = {}
with open(path) as fh:
exec(compile(fh.read(), path, "exec"), ns) # noqa: S102 (trusted, in-repo)
deps = ns.get("DEPS") or []
return [str(d) for d in deps if d]
def dep_domain(parent_recipe: str, pr: str, ref: str | None, dep_recipe: str) -> str: def dep_domain(parent_recipe: str, pr: str, ref: str | None, dep_recipe: str) -> str:
@ -62,11 +50,11 @@ def write_run_state(deps_state) -> None:
"""Write the deps state file ($CCCI_DEPS_FILE). Two shapes supported (canonical=keyed dict): """Write the deps state file ($CCCI_DEPS_FILE). Two shapes supported (canonical=keyed dict):
1. **Legacy list-of-entries:** `[{"recipe": "<dep>", "domain": "<d>"}, ...]` (Q2.3 original). 1. **Legacy list-of-entries:** `[{"recipe": "<dep>", "domain": "<d>"}, ...]` (Q2.3 original).
Still accepted by `load_run_state` for backwards compat — `deps_apps` fixture flattens. Still accepted by `load_run_state` for backwards compat — the `deps` fixture flattens.
2. **NEW per-spec dict (operator-2026-05-28 SSO-dep plan §3.2):** 2. **NEW per-spec dict (operator-2026-05-28 SSO-dep plan §3.2):**
`{"<dep_recipe>": {"recipe": "<dep>", "domain": "<d>", "realm": "...", `{"<dep_recipe>": {"recipe": "<dep>", "domain": "<d>", "realm": "...",
"client_id": "...", "client_secret": "...", "admin_user": "...", "admin_password": "..."}}`. "client_id": "...", "client_secret": "...", "admin_user": "...", "admin_password": "..."}}`.
The `setup_custom_tests.sh` per-recipe hook reads this via `jq` to wire OIDC env. The per-recipe `install_steps.sh` hook reads this via `jq` to wire OIDC env.
No-op if `$CCCI_DEPS_FILE` isn't set.""" No-op if `$CCCI_DEPS_FILE` isn't set."""
path = os.environ.get("CCCI_DEPS_FILE") path = os.environ.get("CCCI_DEPS_FILE")
@ -81,11 +69,12 @@ def deploy_deps(
pr: str, pr: str,
ref: str | None, ref: str | None,
deps: Iterable[str], deps: Iterable[str],
meta_for: dict[str, dict] | None = None, meta_for: dict | None = None,
) -> list[dict]: ) -> list[dict]:
"""Deploy each declared dep, sequentially, at its per-run domain. Returns the list of state """Deploy each declared dep, sequentially, at its per-run domain. Returns the list of state
dicts (one per dep). `meta_for` maps dep_recipe -> meta (HEALTH_PATH/HEALTH_OK/timeouts) so the dicts (one per dep). `meta_for` maps dep_recipe -> RecipeMeta (HEALTH_PATH/HEALTH_OK/timeouts)
readiness wait uses per-dep config; missing dep meta falls back to (/, 200/301/302, 600s).""" so the readiness wait uses per-dep config; a missing dep meta is loaded via meta.load()
(defaults: /, 200/301/302, 600s)."""
meta_for = meta_for or {} meta_for = meta_for or {}
state: list[dict] = [] state: list[dict] = []
for dep in deps: for dep in deps:
@ -94,20 +83,21 @@ def deploy_deps(
# NB: each dep_app gets a fresh deploy_count entry only on `_record_deploy` which fires # NB: each dep_app gets a fresh deploy_count entry only on `_record_deploy` which fires
# inside `lifecycle.deploy_app`. For Phase 2 the deploy-count guard (DG4.1) counts the # inside `lifecycle.deploy_app`. For Phase 2 the deploy-count guard (DG4.1) counts the
# parent + its deps as distinct install events — by design, since each is a separate app. # parent + its deps as distinct install events — by design, since each is a separate app.
dm = meta_for.get(dep, {}) dm = meta_for.get(dep) or meta_mod.load(dep)
lifecycle.deploy_app( lifecycle.deploy_app(
dep, dep,
domain, domain,
secrets=True, secrets=True,
deploy_timeout=int(dm.get("DEPLOY_TIMEOUT", 900)), deploy_timeout=int(dm.DEPLOY_TIMEOUT),
meta=dm,
) )
try: try:
lifecycle.wait_healthy( lifecycle.wait_healthy(
domain, domain,
ok_codes=tuple(dm.get("HEALTH_OK", (200, 301, 302))), ok_codes=tuple(dm.HEALTH_OK),
path=dm.get("HEALTH_PATH", "/"), path=dm.HEALTH_PATH,
deploy_timeout=int(dm.get("DEPLOY_TIMEOUT", 600)), deploy_timeout=int(dm.DEPLOY_TIMEOUT),
http_timeout=int(dm.get("HTTP_TIMEOUT", 600)), http_timeout=int(dm.HTTP_TIMEOUT),
) )
except Exception: except Exception:
# If a dep fails to converge, abort the whole resolve — let the caller teardown # If a dep fails to converge, abort the whole resolve — let the caller teardown
@ -163,7 +153,7 @@ def load_run_state():
def deps_as_dict(state) -> dict[str, dict]: def deps_as_dict(state) -> dict[str, dict]:
"""Coerce either shape (legacy list or new dict) into a recipe→entry dict for the deps_apps """Coerce either shape (legacy list or new dict) into a recipe→entry dict for the `deps`
fixture + dependent-tests consumption.""" fixture + dependent-tests consumption."""
if isinstance(state, dict): if isinstance(state, dict):
return state return state

28
runner/harness/discovery.py
View File

@ -11,7 +11,8 @@ hook; the orchestrator decides additive-vs-skip. Sources, in precedence order
> cc-ci tests/<recipe>/test_<op>.py > cc-ci tests/<recipe>/test_<op>.py
(the generic tests/_generic/test_<op>.py is the always-present floor, run separately by default) (the generic tests/_generic/test_<op>.py is the always-present floor, run separately by default)
custom (non-lifecycle) test_*.py — ALL run, additively, from BOTH locations (opt-in). custom test_*.py (functional/ + playwright/ ONLY, rcust P4 placement rule) — ALL run,
additively, from BOTH locations (opt-in).
install-steps hook — install_steps.sh: repo-local > cc-ci, or none. install-steps hook — install_steps.sh: repo-local > cc-ci, or none.
@ -100,29 +101,22 @@ def resolve_op(recipe: str, op: str, repo_local_dir: str | None) -> tuple[str, s
def custom_tests(recipe: str, repo_local_dir: str | None) -> list[tuple[str, str]]: def custom_tests(recipe: str, repo_local_dir: str | None) -> list[tuple[str, str]]:
"""All non-lifecycle test_*.py from cc-ci's tests/<recipe>/ and (if approved) the recipe's """All custom-tier test_*.py from cc-ci's tests/<recipe>/ and (if approved) the recipe's
repo-local tests/. Discovered locations (Phase 2 §4.1): repo-local tests/. PLACEMENT RULE (rcust P4): custom tests live ONLY under
- the top-level dir tests/<recipe>/test_*.py (legacy + cross-cutting) - functional/ tests/<recipe>/functional/test_*.py (parity ports + recipe-specific)
- functional/ tests/<recipe>/functional/test_*.py (parity ports + recipe-specific) - playwright/ tests/<recipe>/playwright/test_*.py (UI flows)
- playwright/ tests/<recipe>/playwright/test_*.py (UI flows P6) A top-level test_*.py is a LIFECYCLE OVERLAY (test_<op>.py) and nothing else — top-level
Files named `test_<op>.py` (lifecycle ops) are excluded from this list — the orchestrator runs non-lifecycle files are NOT discovered (zero users at the time of the change; the lifecycle-
those in their lifecycle tier, not the custom one. Repo-local is consulted only for name exclusion below stays as a safety net so a misfiled test_<op>.py can never double-run).
allowlist-approved recipes (HC2).""" Repo-local is consulted only for allowlist-approved recipes (HC2)."""
lifecycle_names = {f"test_{op}.py" for op in LIFECYCLE_OPS} lifecycle_names = {f"test_{op}.py" for op in LIFECYCLE_OPS}
subdirs = ("functional", "playwright") subdirs = ("functional", "playwright")
found: list[tuple[str, str]] = [] found: list[tuple[str, str]] = []
for source, d in (("cc-ci", cc_ci_dir(recipe)), ("repo-local", _gated(recipe, repo_local_dir))): for source, d in (("cc-ci", cc_ci_dir(recipe)), ("repo-local", _gated(recipe, repo_local_dir))):
if not d or not os.path.isdir(d): if not d or not os.path.isdir(d):
continue continue
# top-level (legacy / cross-cutting tests not under functional/playwright)
for p in sorted(glob.glob(os.path.join(d, "test_*.py"))):
if os.path.basename(p) not in lifecycle_names:
found.append((source, p))
# functional/ and playwright/ subdirs (Phase 2 §4.1)
for sub in subdirs: for sub in subdirs:
for p in sorted(glob.glob(os.path.join(d, sub, "test_*.py"))): for p in sorted(glob.glob(os.path.join(d, sub, "test_*.py"))):
# Phase-2 layout: lifecycle ops never live under functional/playwright, but be
# explicit so a misfiled file doesn't silently get double-run.
if os.path.basename(p) not in lifecycle_names: if os.path.basename(p) not in lifecycle_names:
found.append((source, p)) found.append((source, p))
return found return found
@ -144,7 +138,7 @@ def install_steps(recipe: str, repo_local_dir: str | None) -> tuple[str, str] |
def pre_op_hook(recipe: str, op: str, repo_local_dir: str | None) -> tuple[str, str] | None: def pre_op_hook(recipe: str, op: str, repo_local_dir: str | None) -> tuple[str, str] | None:
"""The pre-op seed hook for `op`: the path to a recipe `ops.py` module that defines a """The pre-op seed hook for `op`: the path to a recipe `ops.py` module that defines a
`pre_<op>(domain, meta)` callable, or None. cc-ci's tests/<recipe>/ops.py wins; the repo-local `pre_<op>(ctx)` callable, or None. cc-ci's tests/<recipe>/ops.py wins; the repo-local
ops.py is consulted only for allowlist-approved recipes (HC2). The orchestrator imports the ops.py is consulted only for allowlist-approved recipes (HC2). The orchestrator imports the
module and calls pre_<op> BEFORE performing the op (HC3 op/assertion split — overlays seed module and calls pre_<op> BEFORE performing the op (HC3 op/assertion split — overlays seed
pre-op state here, then assert post-op in test_<op>.py).""" pre-op state here, then assert post-op in test_<op>.py)."""

47
runner/harness/generic.py
View File

@ -19,6 +19,7 @@ import ssl
import time import time
from . import abra, lifecycle from . import abra, lifecycle
from . import meta as meta_mod
# A recipe is backup-capable iff a compose file carries a truthy backupbot.backup label. # A recipe is backup-capable iff a compose file carries a truthy backupbot.backup label.
_BACKUPBOT_RE = re.compile(r"backupbot\.backup\b[^\n]*\btrue\b", re.IGNORECASE) _BACKUPBOT_RE = re.compile(r"backupbot\.backup\b[^\n]*\btrue\b", re.IGNORECASE)
@ -28,13 +29,14 @@ def _recipe_dir(recipe: str) -> str:
return abra.recipe_dir(recipe) # the per-run tree inside a CI run ($ABRA_DIR) return abra.recipe_dir(recipe) # the per-run tree inside a CI run ($ABRA_DIR)
def backup_capable(recipe: str, meta: dict | None = None) -> bool: def backup_capable(recipe: str, meta=None) -> bool:
"""Whether the harness should run the backup/restore tiers (else they are a clean N/A skip, DG3). """Whether the harness should run the backup/restore tiers (else they are a clean N/A skip, DG3).
`recipe_meta.BACKUP_CAPABLE` (bool) overrides; otherwise auto-detect by scanning the recipe's `recipe_meta.BACKUP_CAPABLE` (bool) overrides when explicitly set (RecipeMeta default is None =
compose*.yml for a truthy `backupbot.backup` label (the Co-op Cloud backup convention).""" unset); otherwise auto-detect by scanning the recipe's compose*.yml for a truthy
if meta and "BACKUP_CAPABLE" in meta: `backupbot.backup` label (the Co-op Cloud backup convention)."""
return bool(meta["BACKUP_CAPABLE"]) if meta is not None and meta.BACKUP_CAPABLE is not None:
return bool(meta.BACKUP_CAPABLE)
for path in glob.glob(os.path.join(_recipe_dir(recipe), "compose*.yml")): for path in glob.glob(os.path.join(_recipe_dir(recipe), "compose*.yml")):
try: try:
with open(path) as fh: with open(path) as fh:
@ -75,7 +77,7 @@ def served_cert(domain: str, port: int = 443) -> tuple[bool, str]:
return (True, f"CN={cn} SAN={sans}") return (True, f"CN={cn} SAN={sans}")
def assert_serving(domain: str, meta: dict) -> None: def assert_serving(domain: str, meta) -> None:
"""The single generic "is the app really serving?" assertion (DG1). """The single generic "is the app really serving?" assertion (DG1).
The app-vs-Traefik-fallback proof is steps 1+2 (both load-bearing, verified by the Adversary): The app-vs-Traefik-fallback proof is steps 1+2 (both load-bearing, verified by the Adversary):
@ -90,14 +92,14 @@ def assert_serving(domain: str, meta: dict) -> None:
Steps 1–2 are BOUNDED POLLS (no bare sleep), so a state-mutating op (upgrade/restore) that leaves Steps 1–2 are BOUNDED POLLS (no bare sleep), so a state-mutating op (upgrade/restore) that leaves
the app briefly reconverging settles, while a persistent failure still fails within the timeout.""" the app briefly reconverging settles, while a persistent failure still fails within the timeout."""
deadline = time.time() + meta["DEPLOY_TIMEOUT"] deadline = time.time() + meta.DEPLOY_TIMEOUT
while time.time() < deadline and not lifecycle.services_converged(domain): while time.time() < deadline and not lifecycle.services_converged(domain):
time.sleep(5) time.sleep(5)
assert lifecycle.services_converged(domain), f"{domain}: services did not converge" assert lifecycle.services_converged(domain), f"{domain}: services did not converge"
path = meta["HEALTH_PATH"] path = meta.HEALTH_PATH
ok = tuple(meta["HEALTH_OK"]) ok = tuple(meta.HEALTH_OK)
deadline = time.time() + meta["HTTP_TIMEOUT"] deadline = time.time() + meta.HTTP_TIMEOUT
served = False served = False
status, body = 0, "" status, body = 0, ""
while time.time() < deadline: while time.time() < deadline:
@ -141,7 +143,7 @@ def op_state() -> dict:
return {} return {}
def assert_upgraded(domain: str, meta: dict) -> None: def assert_upgraded(domain: str, meta) -> None:
"""Generic UPGRADE assertion (post-op): the orchestrator already performed the upgrade once via """Generic UPGRADE assertion (post-op): the orchestrator already performed the upgrade once via
`abra app deploy --chaos` of the PR-head checkout. Assert it reconverged + still serves AND that `abra app deploy --chaos` of the PR-head checkout. Assert it reconverged + still serves AND that
the deployment is genuinely the PR-head code under test (HC1) — non-vacuously (guarding F1d-2). the deployment is genuinely the PR-head code under test (HC1) — non-vacuously (guarding F1d-2).
@ -212,7 +214,7 @@ def assert_backup_artifact(domain: str) -> str:
return snap_id return snap_id
def assert_restore_healthy(domain: str, meta: dict) -> None: def assert_restore_healthy(domain: str, meta) -> None:
"""Generic RESTORE assertion (post-op): the orchestrator already restored. Assert the app is """Generic RESTORE assertion (post-op): the orchestrator already restored. Assert the app is
healthy + serving again (assert_serving polls, so the post-restore reconverge settles).""" healthy + serving again (assert_serving polls, so the post-restore reconverge settles)."""
assert_serving(domain, meta) assert_serving(domain, meta)
@ -226,7 +228,7 @@ def perform_upgrade(
recipe: str, recipe: str,
head_ref: str | None, head_ref: str | None,
deploy_timeout: int = 900, deploy_timeout: int = 900,
meta: dict | None = None, meta=None,
) -> dict[str, str | None]: ) -> dict[str, str | None]:
"""Perform the UPGRADE op once, in place, to the PR-HEAD code under test (HC1): re-checkout the """Perform the UPGRADE op once, in place, to the PR-HEAD code under test (HC1): re-checkout the
PR head (the prev-tag base deploy reset the recipe working tree), then `abra app deploy --chaos` PR head (the prev-tag base deploy reset the recipe working tree), then `abra app deploy --chaos`
@ -244,7 +246,8 @@ def perform_upgrade(
STRICTER convergence+health wait here: services N/N (wait_healthy) + app HEALTH_PATH healthy + STRICTER convergence+health wait here: services N/N (wait_healthy) + app HEALTH_PATH healthy +
any recipe READY_PROBE (collabora WOPI discovery 200). This bounds readiness by OUR generous any recipe READY_PROBE (collabora WOPI discovery 200). This bounds readiness by OUR generous
deadline, not abra's impatient one — and is stronger evidence than abra's monitor.""" deadline, not abra's impatient one — and is stronger evidence than abra's monitor."""
meta = meta or {} if meta is None:
meta = meta_mod.load(recipe)
before = lifecycle.deployed_identity(domain) before = lifecycle.deployed_identity(domain)
if head_ref: if head_ref:
lifecycle.recipe_checkout_ref(recipe, head_ref) lifecycle.recipe_checkout_ref(recipe, head_ref)
@ -253,9 +256,7 @@ def perform_upgrade(
# (target) version, so the base deploys minimally WITHOUT it and the upgrade adds it to COMPOSE_FILE # (target) version, so the base deploys minimally WITHOUT it and the upgrade adds it to COMPOSE_FILE
# here, after the PR-head checkout (which ships the overlay) and before the chaos redeploy that # here, after the PR-head checkout (which ships the overlay) and before the chaos redeploy that
# picks up the new .env. Dict or callable(domain)->dict. No-op for recipes without it. # picks up the new .env. Dict or callable(domain)->dict. No-op for recipes without it.
upgrade_env = meta.get("UPGRADE_EXTRA_ENV") or {} upgrade_env = meta_mod.upgrade_extra_env(meta, meta_mod.hook_ctx(domain, meta, op="upgrade"))
if callable(upgrade_env):
upgrade_env = upgrade_env(domain) or {}
for k, v in upgrade_env.items(): for k, v in upgrade_env.items():
print(f" upgrade-env: {k}={v}", flush=True) print(f" upgrade-env: {k}={v}", flush=True)
abra.env_set(domain, k, v) abra.env_set(domain, k, v)
@ -266,14 +267,12 @@ def perform_upgrade(
# Own the convergence verification (abra's monitor was skipped via -c). # Own the convergence verification (abra's monitor was skipped via -c).
lifecycle.wait_healthy( lifecycle.wait_healthy(
domain, domain,
ok_codes=tuple(meta.get("HEALTH_OK", (200, 301, 302))), ok_codes=tuple(meta.HEALTH_OK),
path=meta.get("HEALTH_PATH", "/"), path=meta.HEALTH_PATH,
deploy_timeout=int(meta.get("DEPLOY_TIMEOUT", deploy_timeout)), deploy_timeout=int(meta.DEPLOY_TIMEOUT),
http_timeout=int(meta.get("HTTP_TIMEOUT", 300)), http_timeout=int(meta.HTTP_TIMEOUT),
)
lifecycle.wait_ready_probes(
meta, domain, timeout=int(meta.get("DEPLOY_TIMEOUT", deploy_timeout))
) )
lifecycle.wait_ready_probes(meta, domain, timeout=int(meta.DEPLOY_TIMEOUT), op="upgrade")
after = lifecycle.deployed_identity(domain) after = lifecycle.deployed_identity(domain)
# Evidence (HC1): the chaos-version label = the deployed recipe commit; it should match the # Evidence (HC1): the chaos-version label = the deployed recipe commit; it should match the
# PR-head we checked out — proving the upgrade deployed the code under test, not a published tag. # PR-head we checked out — proving the upgrade deployed the code under test, not a published tag.

105
runner/harness/lifecycle.py
View File

@ -12,6 +12,7 @@ import glob
import json import json
import os import os
import re import re
import shutil
import socket import socket
import ssl import ssl
import subprocess import subprocess
@ -19,6 +20,7 @@ import time
import urllib.request import urllib.request
from . import abra, lifetime from . import abra, lifetime
from . import meta as meta_mod
GATEWAY_IP = "143.244.213.108" # *.ci.commoninternet.net -> gateway (TLS passthrough to cc-ci) GATEWAY_IP = "143.244.213.108" # *.ci.commoninternet.net -> gateway (TLS passthrough to cc-ci)
# A run app domain is "<recipe[:4]>-<6hex>.ci.commoninternet.net" (see DECISIONS.md). Used by the # A run app domain is "<recipe[:4]>-<6hex>.ci.commoninternet.net" (see DECISIONS.md). Used by the
@ -111,37 +113,6 @@ def _residual(domain: str) -> dict:
} }
def _recipe_extra_env(recipe: str, domain: str) -> dict[str, str]:
"""Per-recipe extra .env keys, applied at every deploy (install + upgrade's old_app) so a recipe
with multi-domain / config needs is enrolled with NO shared-harness change (D5/M6.5). A recipe
declares `EXTRA_ENV` in tests/<recipe>/recipe_meta.py as either a dict or a callable
`EXTRA_ENV(domain) -> dict` (callable form lets it derive values from the per-run domain, e.g.
cryptpad's SANDBOX_DOMAIN). Returns {} if none."""
path = os.path.join(os.path.dirname(__file__), "..", "..", "tests", recipe, "recipe_meta.py")
if not os.path.exists(path):
return {}
ns: dict = {}
with open(path) as fh:
exec(compile(fh.read(), path, "exec"), ns) # noqa: S102 (trusted, in-repo)
ee = ns.get("EXTRA_ENV")
if callable(ee):
ee = ee(domain)
return {str(k): str(v) for k, v in (ee or {}).items()}
def _recipe_meta_flag(recipe: str, key: str) -> bool:
"""Read a boolean flag from tests/<recipe>/recipe_meta.py (e.g. CHAOS_BASE_DEPLOY). Returns
False if the recipe ships no meta or the flag is absent/falsey. Trusted in-repo exec, same as
_recipe_extra_env."""
path = os.path.join(os.path.dirname(__file__), "..", "..", "tests", recipe, "recipe_meta.py")
if not os.path.exists(path):
return False
ns: dict = {}
with open(path) as fh:
exec(compile(fh.read(), path, "exec"), ns) # noqa: S102 (trusted, in-repo)
return bool(ns.get(key))
def _record_deploy() -> None: def _record_deploy() -> None:
"""Increment the per-run deploy counter (DG4.1: one deploy per run). No-op unless the """Increment the per-run deploy counter (DG4.1: one deploy per run). No-op unless the
orchestrator set CCCI_DEPLOY_COUNT_FILE — so it never affects standalone/manual use.""" orchestrator set CCCI_DEPLOY_COUNT_FILE — so it never affects standalone/manual use."""
@ -155,6 +126,34 @@ def _record_deploy() -> None:
f.write(str(n + 1)) f.write(str(n + 1))
def ccci_overlay_path(recipe: str) -> str:
"""The cc-ci-owned compose overlay for a recipe (rcust P2a: first-class, auto-discovered)."""
return os.path.join(meta_mod.TESTS_DIR, recipe, "compose.ccci.yml")
def has_ccci_overlay(recipe: str) -> bool:
return os.path.isfile(ccci_overlay_path(recipe))
def provide_ccci_overlay(recipe: str) -> None:
"""Copy tests/<recipe>/compose.ccci.yml into THIS run's recipe checkout (ABRA_DIR-aware), so
the recipe's COMPOSE_FILE reference resolves (rcust P2a — the harness owns the copy; recipes
no longer ship install_steps.sh boilerplate for it). No-op for recipes without an overlay."""
src = ccci_overlay_path(recipe)
if not os.path.isfile(src):
return
dest_dir = abra.recipe_dir(recipe)
if not os.path.isdir(dest_dir):
print(f" ccci-overlay: recipe dir {dest_dir} missing — cannot provide overlay", flush=True)
raise RuntimeError(f"recipe checkout missing for {recipe}: {dest_dir}")
shutil.copy(src, os.path.join(dest_dir, "compose.ccci.yml"))
print(
f" ccci-overlay: provided compose.ccci.yml to the {recipe} checkout "
"(first-class overlay; base deploy auto-chaos)",
flush=True,
)
def _run_install_steps(hook: tuple[str, str], recipe: str, domain: str) -> None: def _run_install_steps(hook: tuple[str, str], recipe: str, domain: str) -> None:
"""Run a recipe's custom install-steps hook (install_steps.sh) during the install tier — after """Run a recipe's custom install-steps hook (install_steps.sh) during the install tier — after
`abra app new` + env defaults + secret generate, before deploy (Phase 1d DG5). The hook gets the `abra app new` + env defaults + secret generate, before deploy (Phase 1d DG5). The hook gets the
@ -238,15 +237,23 @@ def deploy_app(
secrets: bool = True, secrets: bool = True,
install_steps_hook: tuple[str, str] | None = None, install_steps_hook: tuple[str, str] | None = None,
deploy_timeout: int = 900, deploy_timeout: int = 900,
meta=None,
) -> None: ) -> None:
"""Create + configure + deploy an app. Forces LETS_ENCRYPT_ENV='' so traefik serves the """Create + configure + deploy an app. Forces LETS_ENCRYPT_ENV='' so traefik serves the
wildcard cert via the file provider and NEVER attempts ACME (adversary finding A1). Applies any wildcard cert via the file provider and NEVER attempts ACME (adversary finding A1). Applies any
per-recipe EXTRA_ENV (recipe_meta.py) and the custom install-steps hook (Phase 1d) before deploy. per-recipe EXTRA_ENV (recipe_meta.py), the custom install-steps hook (Phase 1d), and the
first-class `tests/<recipe>/compose.ccci.yml` overlay (rcust P2a) before deploy.
`meta` is the recipe's loaded RecipeMeta (EXTRA_ENV); the orchestrator loads once and passes
it down. Callers without one in hand (fixtures, warm reconcile) may omit it — it is then
loaded here via the single meta.load() path.
`deploy_timeout` is the subprocess timeout for `abra app deploy`. Caller (orchestrator) passes `deploy_timeout` is the subprocess timeout for `abra app deploy`. Caller (orchestrator) passes
`recipe_meta.DEPLOY_TIMEOUT` so heavy recipes (ghost, matrix-synapse, lasuite-meet) can extend `recipe_meta.DEPLOY_TIMEOUT` so heavy recipes (ghost, matrix-synapse, lasuite-meet) can extend
past the 900s default. abra's INTERNAL TIMEOUT (recipe's TIMEOUT env, default 300s) is set via past the 900s default. abra's INTERNAL TIMEOUT (recipe's TIMEOUT env, default 300s) is set via
EXTRA_ENV; this is the Python subprocess wrapper's timeout so abra doesn't get SIGKILLed mid-deploy.""" EXTRA_ENV; this is the Python subprocess wrapper's timeout so abra doesn't get SIGKILLed mid-deploy."""
if meta is None:
meta = meta_mod.load(recipe)
_record_deploy() _record_deploy()
# Lock BEFORE the app exists: a concurrent run's janitor must never see this app without a # Lock BEFORE the app exists: a concurrent run's janitor must never see this app without a
# held app lock (it would probe it as an orphan and reap an in-flight deploy). Also the # held app lock (it would probe it as an orphan and reap an in-flight deploy). Also the
@ -274,16 +281,18 @@ def deploy_app(
flush=True, flush=True,
) )
chaos = True chaos = True
# A recipe may force a chaos base deploy via recipe_meta CHAOS_BASE_DEPLOY=True when an # A first-class cc-ci compose overlay (tests/<recipe>/compose.ccci.yml, copied into the
# install_steps hook adds an untracked compose overlay to the recipe checkout (e.g. discourse's # checkout below — rcust P2a) is an UNTRACKED file in the recipe checkout, which makes
# compose.ccci.yml, provided by install_steps for the pinned base). The untracked file makes # abra's pinned-deploy clean-tree check FATA ('has locally unstaged changes'). Auto-chaos:
# abra's pinned-deploy clean-tree check FATA ('has locally unstaged changes'); chaos skips lint + # chaos skips lint + the clean-tree gate and deploys the EXPLICITLY-checked-out pinned
# the clean-tree gate and deploys the EXPLICITLY-checked-out pinned version (we already ran # version (we already ran recipe_checkout(version) above) — NOT latest. Same mechanism as
# recipe_checkout(version) above) — NOT latest. Same mechanism as the lightweight-tag branch. # the lightweight-tag branch. (Replaces the deleted CHAOS_BASE_DEPLOY meta flag — the
elif _recipe_meta_flag(recipe, "CHAOS_BASE_DEPLOY"): # overlay's presence IS the signal, killing the R7 implicit coupling.)
elif has_ccci_overlay(recipe):
print( print(
f" deploy_app({recipe}@{version}): CHAOS_BASE_DEPLOY set → chaos base deploy of the " f" deploy_app({recipe}@{version}): compose.ccci.yml overlay present → chaos base "
"checked-out pinned version (skips clean-tree/lint; deploys version, not LATEST)", "deploy of the checked-out pinned version (skips clean-tree/lint; deploys version, "
"not LATEST)",
flush=True, flush=True,
) )
chaos = True chaos = True
@ -293,12 +302,18 @@ def deploy_app(
# it ourselves is recipe-agnostic and canonical (the run domain IS the app's domain). # it ourselves is recipe-agnostic and canonical (the run domain IS the app's domain).
abra.env_set(domain, "DOMAIN", domain) abra.env_set(domain, "DOMAIN", domain)
abra.env_set(domain, "LETS_ENCRYPT_ENV", "") abra.env_set(domain, "LETS_ENCRYPT_ENV", "")
for k, v in _recipe_extra_env(recipe, domain).items(): for k, v in meta_mod.extra_env(meta, meta_mod.hook_ctx(domain, meta)).items():
abra.env_set(domain, k, v) abra.env_set(domain, k, v)
if secrets: if secrets:
abra.secret_generate(domain) abra.secret_generate(domain)
if install_steps_hook: if install_steps_hook:
_run_install_steps(install_steps_hook, recipe, domain) _run_install_steps(install_steps_hook, recipe, domain)
# First-class cc-ci compose overlay (rcust P2a): if the recipe ships
# tests/<recipe>/compose.ccci.yml, copy it into THIS run's recipe checkout (ABRA_DIR-aware)
# so the COMPOSE_FILE reference in the recipe's EXTRA_ENV resolves. Untracked, so it persists
# across the later PR-head checkout (idempotent when the head ships the same fix). Replaces
# the per-recipe install_steps.sh copy boilerplate + CHAOS_BASE_DEPLOY flag (auto-chaos above).
provide_ccci_overlay(recipe)
# HQ1: warm the local image store before the (real, unchanged) abra deploy. # HQ1: warm the local image store before the (real, unchanged) abra deploy.
prepull_images(recipe, domain) prepull_images(recipe, domain)
abra.deploy(domain, chaos=chaos, timeout=deploy_timeout) abra.deploy(domain, chaos=chaos, timeout=deploy_timeout)
@ -510,7 +525,7 @@ def chaos_redeploy(
abra.deploy(domain, chaos=True, timeout=deploy_timeout, no_converge_checks=no_converge_checks) abra.deploy(domain, chaos=True, timeout=deploy_timeout, no_converge_checks=no_converge_checks)
def wait_ready_probes(meta: dict, domain: str, timeout: int = 600) -> None: def wait_ready_probes(meta, domain: str, timeout: int = 600, op: str | None = None) -> None:
"""Poll a recipe's optional READY_PROBE endpoints until each returns an accepted status, or raise. """Poll a recipe's optional READY_PROBE endpoints until each returns an accepted status, or raise.
A recipe_meta may define `READY_PROBE(domain) -> [{"host":..., "path":..., "ok":(200,)}, ...]` A recipe_meta may define `READY_PROBE(domain) -> [{"host":..., "path":..., "ok":(200,)}, ...]`
@ -527,10 +542,10 @@ def wait_ready_probes(meta: dict, domain: str, timeout: int = 600) -> None:
must be released by the old task + rebound by the new) the voice server can be down while must be released by the old task + rebound by the new) the voice server can be down while
HTTP-200 still passes — and backup-bot then execs into a not-running app container (409). Requiring HTTP-200 still passes — and backup-bot then execs into a not-running app container (409). Requiring
the voice port to be stably listening before proceeding closes that window.""" the voice port to be stably listening before proceeding closes that window."""
probe_fn = meta.get("READY_PROBE") probe_fn = meta.READY_PROBE
if not callable(probe_fn): if not callable(probe_fn):
return return
probes = probe_fn(domain) or [] probes = probe_fn(meta_mod.hook_ctx(domain, meta, op=op)) or []
for probe in probes: for probe in probes:
if "tcp_port" in probe: if "tcp_port" in probe:
host = probe.get("tcp_host", "127.0.0.1") host = probe.get("tcp_host", "127.0.0.1")

153
runner/harness/manifest.py Normal file
View File

@ -0,0 +1,153 @@
"""Customization manifest (rcust P5; spec §8 R4 mitigation).
One block at run start answering "what does this recipe customize?" across ALL the surfaces
(recipe_meta keys, hook files, file-presence, run-time env overrides) — printed to the run log and
embedded verbatim in results.json under "customization". PURE PRESENTATION: building or printing
the manifest must never influence any verdict (R7-class invariant).
"""
from __future__ import annotations
import os
import re
from . import discovery, lifecycle
from . import meta as meta_mod
_PRE_OP_RE = re.compile(r"^def (pre_[a-z]+)\(", re.MULTILINE)
# Meta values are repo-public by construction (recipe_meta.py is committed; real secrets are
# class-B generated, never meta), but the manifest lands on the dashboard — mask values whose
# key NAME is secret-shaped so a field literally called SECRET_KEY_BASE never shows a value
# (defense in depth + keeps dashboard secret-scans quiet). `KEY` matches only as a word segment
# (API_KEY yes, KEYCLOAK_URL no).
_SENSITIVE_NAME_RE = re.compile(r"SECRET|PASSWORD|TOKEN|CREDENTIAL|(^|_)KEY(_|$)", re.IGNORECASE)
def _jsonable(v, name=""):
"""Manifest values must be JSON-serializable + deterministic: hooks render as '<hook>',
tuples become lists, secret-named entries (by key name, incl. nested dict keys) as
'<redacted>'."""
if callable(v):
return "<hook>"
if name and _SENSITIVE_NAME_RE.search(name):
return "<redacted>"
if isinstance(v, tuple):
return list(v)
if isinstance(v, dict):
return {k: _jsonable(x, name=str(k)) for k, x in v.items()}
return v
def _pre_ops(path: str) -> list[str]:
"""The pre_<op> hook names an ops.py defines (cheap source scan, same approach as
discovery._module_defines — no import)."""
try:
with open(path) as fh:
return sorted(set(_PRE_OP_RE.findall(fh.read())))
except OSError:
return []
def _custom_counts(recipe: str, repo_local: str | None) -> dict[str, dict[str, int]]:
out: dict[str, dict[str, int]] = {}
for source, path in discovery.custom_tests(recipe, repo_local):
sub = os.path.basename(os.path.dirname(path)) # functional | playwright
out.setdefault(source, {}).setdefault(sub, 0)
out[source][sub] += 1
return out
def build(recipe: str, meta, repo_local: str | None) -> dict:
"""Collect the run's resolved customization into one deterministic, JSON-serializable dict.
Keys: meta_non_default (explicitly-customized recipe_meta keys), hooks (ops.py pre-ops +
install_steps.sh + compose.ccci.yml with their source), overlays (lifecycle overlay files by
op + source), custom_tests (counts per source/subdir), env_overrides (active
CCCI_SKIP_GENERIC* — the dev-only escape hatch, flagged when riding a CI run)."""
hooks: dict = {}
pre_ops: dict[str, list[str]] = {}
for source, d in (
("cc-ci", discovery.cc_ci_dir(recipe)),
("repo-local", discovery._gated(recipe, repo_local)), # noqa: SLF001 — same HC2 gate
):
if not d:
continue
p = os.path.join(d, "ops.py")
if os.path.isfile(p):
ops = _pre_ops(p)
if ops:
pre_ops[source] = ops
if pre_ops:
hooks["ops.py"] = pre_ops
ist = discovery.install_steps(recipe, repo_local)
if ist:
hooks["install_steps.sh"] = ist[0]
if lifecycle.has_ccci_overlay(recipe):
hooks["compose.ccci.yml"] = "cc-ci"
overlays = {}
for op in discovery.LIFECYCLE_OPS:
ov = discovery.resolve_overlay_op(recipe, op, repo_local)
if ov:
overlays[op] = ov[0]
env_overrides = sorted(
k
for k in os.environ
if k.startswith("CCCI_SKIP_GENERIC")
and str(os.environ.get(k) or "").strip().lower() in ("1", "true", "yes", "on")
)
return {
"meta_non_default": {
k: _jsonable(v, name=k) for k, v in sorted(meta_mod.non_default(meta).items())
},
"hooks": hooks,
"overlays": overlays,
"custom_tests": _custom_counts(recipe, repo_local),
"env_overrides": env_overrides,
}
def render(recipe: str, manifest: dict) -> str:
"""The human block printed at run start (same content as the results.json key)."""
lines = [f"===== customization manifest: {recipe} ====="]
nd = manifest["meta_non_default"]
lines.append(
"meta (non-default): "
+ (" ".join(f"{k}={v!r}" for k, v in nd.items()) if nd else "(none — zero-config floor)")
)
hk = manifest["hooks"]
parts = []
for source, ops in hk.get("ops.py", {}).items():
parts.append(f"ops.py[{','.join(ops)}]({source})")
if "install_steps.sh" in hk:
parts.append(f"install_steps.sh({hk['install_steps.sh']})")
if "compose.ccci.yml" in hk:
parts.append(f"compose.ccci.yml({hk['compose.ccci.yml']})")
lines.append("hooks: " + (" ".join(parts) if parts else "(none)"))
ov = manifest["overlays"]
lines.append(
"overlays: "
+ (" ".join(f"test_{op}.py({src})" for op, src in ov.items()) if ov else "(none)")
)
ct = manifest["custom_tests"]
lines.append(
"custom tests: "
+ (
" ".join(
" ".join(f"{sub}/={n}" for sub, n in sorted(counts.items())) + f" ({source})"
for source, counts in sorted(ct.items())
)
if ct
else "(none)"
)
)
eo = manifest["env_overrides"]
if eo:
suffix = " !! dev-only override active in CI" if os.environ.get("DRONE") else ""
lines.append("env overrides: " + " ".join(f"{k}=1" for k in eo) + suffix)
else:
lines.append("env overrides: (none)")
return "\n".join(lines)

320
runner/harness/meta.py Normal file
View File

@ -0,0 +1,320 @@
"""Single recipe-meta loader + declarative key registry (recipe-custom restructure P1; spec
docs/recipe-customization.md §8 R1).
THE one place `tests/<recipe>/recipe_meta.py` is `exec()`d. Every consumer (orchestrator, pytest
`meta` fixture, deploy env shaping, deps, warm-canonical enrollment, screenshot) reads the ONE
loaded `RecipeMeta` object instead of re-exec'ing the file and cherry-picking keys — that drift
(six divergent loaders, spec §4 L1–L6) is what made `SCREENSHOT` an unreachable knob (R2) and let
key typos silently disable coverage (R6).
Validation (locked decision, recipe-custom-restructure-full-plan.md):
- unknown ALL-CAPS top-level name → MetaError (hard error, fails fast at load; the all-recipes
unit test catches it at PR time). Underscore-prefixed names (`_FOO`) are recipe-private and
exempt; lowercase names (helper functions/imports) are ignored.
- type mismatch → MetaError. Callables are accepted ONLY for hook-typed keys.
The KEYS registry is the single source of truth for the key set: it drives validation, the
RecipeMeta dataclass fields, and the generated reference table in docs/recipe-customization.md §4
(scripts/gen-meta-docs.py; a unit test asserts the committed table matches).
"""
from __future__ import annotations
import copy
import dataclasses
import difflib
import inspect
import json
import os
from collections.abc import Callable
ROOT = os.path.dirname(os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
TESTS_DIR = os.path.join(ROOT, "tests")
class MetaError(Exception):
"""A recipe_meta.py failed registry validation (unknown key / type mismatch / callable on a
data key). Hard error by design: a typo'd key must fail the run at load, not silently reduce
coverage (spec §8 R6 — the worst failure mode for a CI harness)."""
@dataclasses.dataclass(frozen=True)
class Key:
"""One registered recipe_meta key: name, type tag, default, one-line doc (rendered into the
generated reference table), optional extra validator, and a deprecation marker (deprecated
keys still load+validate but are scheduled for deletion)."""
name: str
type: str # "int"|"str"|"tuple[int]"|"bool"|"dict_or_hook"|"hook"|"list[str]"|"dict"
default: object
doc: str
validate: Callable[[object], None] | None = None
deprecated: bool = False
# Expected positional-parameter names for a callable value (rcust P3 uniform ctx convention).
# Enforced at load so a legacy-signature hook (e.g. `def READY_PROBE(domain)`) fails with a
# CLEAR MetaError naming the migration — never a silent TypeError mid-run.
hook_params: tuple[str, ...] | None = None
KEYS: tuple[Key, ...] = (
Key(
"HEALTH_PATH",
"str",
"/",
"Path probed for serving/health checks (deploy wait + generic `assert_serving`).",
),
Key("HEALTH_OK", "tuple[int]", (200, 301, 302), "Acceptable HTTP status codes for health."),
Key("DEPLOY_TIMEOUT", "int", 600, "Max seconds to wait for swarm convergence per deploy."),
Key("HTTP_TIMEOUT", "int", 300, "Max seconds to wait for HTTP health after convergence."),
Key(
"BACKUP_CAPABLE",
"bool",
None,
"Override the backup-tier capability auto-detect (compose `backupbot.backup` labels). `False` forces N/A; `True` forces the tier on; unset = auto-detect.",
),
Key(
"EXPECTED_NA",
"dict",
None,
"Declare an N/A rung intentional: `{rung: reason}`. The cap stands either way; only the report wording changes.",
),
Key(
"READY_PROBE",
"hook",
None,
"Callable `(ctx) -> [probe, ...]` returning extra readiness probes, run after install AND after upgrade: HTTP `{host, path, ok}` or TCP `{tcp_host, tcp_port, stable}`.",
hook_params=("ctx",),
),
Key(
"UPGRADE_BASE_VERSION",
"str",
None,
"Exact published tag overriding the upgrade tier's base (default: `recipe_versions[-2]`).",
),
Key(
"BACKUP_VERIFY",
"hook",
None,
"Callable `(ctx) -> bool` post-backup data-capture check; `False` re-runs the backup (truncated-dump race guard), retried up to 3 attempts.",
hook_params=("ctx",),
),
Key(
"UPGRADE_EXTRA_ENV",
"dict_or_hook",
None,
"Extra `.env` keys applied after the PR-head checkout, before the chaos redeploy (env that exists only at head). Dict, or callable `(ctx) -> dict`.",
hook_params=("ctx",),
),
Key(
"EXTRA_ENV",
"dict_or_hook",
{},
"Extra `.env` keys applied at EVERY deploy (base install AND upgrade old-app). Dict, or callable `(ctx) -> dict` deriving values from the per-run domain (`ctx.domain`).",
hook_params=("ctx",),
),
Key(
"DEPS",
"list[str]",
[],
'Dep recipes deployed/provisioned alongside (e.g. `["keycloak"]`); creds land in `$CCCI_DEPS_FILE`.',
),
Key(
"WARM_CANONICAL",
"bool",
False,
"Enroll the recipe in the warm/canonical app system (docs/warm.md): green cold runs on LATEST advance the canonical snapshot.",
),
Key(
"SCREENSHOT",
"hook",
None,
"Callable `(page, ctx)` driving Playwright to a safe, credential-free post-login view for the results-card screenshot (default: landing page).",
hook_params=("page", "ctx"),
),
# (CHAOS_BASE_DEPLOY, OIDC_AT_INSTALL and SKIP_GENERIC were deleted in restructure P2:
# compose.ccci.yml is first-class + auto-chaos; install-time deps wiring is the only mode;
# the generic floor is suppressible only via the dev-only CCCI_SKIP_GENERIC* env form.)
)
_REGISTRY: dict[str, Key] = {k.name: k for k in KEYS}
# The one validated, attribute-access view of a recipe's customization. Generated from KEYS so the
# field set can never drift from the registry (frozen: consumers share one immutable object).
RecipeMeta = dataclasses.make_dataclass(
"RecipeMeta",
[(k.name, object, dataclasses.field(default=None)) for k in KEYS],
frozen=True,
)
RecipeMeta.__doc__ = (
"Validated per-recipe customization (one field per registered key; attribute access). "
"Built ONLY by meta.load()."
)
def meta_path(recipe: str, tests_dir: str | None = None) -> str:
"""Canonical path of a recipe's meta file (pure)."""
return os.path.join(tests_dir or TESTS_DIR, recipe, "recipe_meta.py")
def check_hook_signature(fn, expected: tuple[str, ...], where: str) -> None:
"""Enforce the uniform ctx hook convention (rcust P3): a hook callable's positional parameters
must be exactly `expected` (e.g. ("ctx",) or ("page", "ctx")). A legacy-signature hook (the
pre-restructure `(domain)` / `(domain, meta)` / `(page, domain, meta)` forms) raises a CLEAR
MetaError naming the migration — never a silent TypeError mid-run."""
try:
params = [
p.name
for p in inspect.signature(fn).parameters.values()
if p.kind in (p.POSITIONAL_ONLY, p.POSITIONAL_OR_KEYWORD)
]
except (TypeError, ValueError): # builtins/odd callables — let the call site surface it
return
if tuple(params) != expected:
raise MetaError(
f"{where}: hook signature is ({', '.join(params)}) — the recipe-customization "
f"restructure (P3) changed ALL recipe hook signatures to ({', '.join(expected)}); "
f"read fields off the HookCtx (ctx.domain, ctx.base_url, ctx.meta, ctx.deps, ctx.op). "
f"See docs/recipe-customization.md §5."
)
def _coerce(key: Key, value: object, path: str) -> object:
"""Validate `value` against `key`'s declared type; normalize containers (tuple[int]/list[str]).
Raises MetaError on mismatch — including a callable supplied for a data-typed key."""
t = key.type
if callable(value) and t not in ("hook", "dict_or_hook"):
raise MetaError(
f"{path}: {key.name} is a data key (type {t}) — callables are accepted only for "
f"hook-typed keys"
)
if t == "int":
if isinstance(value, int) and not isinstance(value, bool):
return value
elif t == "str":
if isinstance(value, str):
return value
elif t == "bool":
if isinstance(value, bool):
return value
elif t == "tuple[int]":
if isinstance(value, tuple | list) and all(
isinstance(x, int) and not isinstance(x, bool) for x in value
):
return tuple(value)
elif t == "list[str]":
if isinstance(value, tuple | list) and all(isinstance(x, str) for x in value):
return list(value)
elif t == "dict":
if isinstance(value, dict):
return value
elif (
t == "hook"
and callable(value)
or t == "dict_or_hook"
and (isinstance(value, dict) or callable(value))
):
return value
raise MetaError(f"{path}: {key.name} must be {t}, got {type(value).__name__} ({value!r})")
def load(recipe: str, tests_dir: str | None = None):
"""Load + validate a recipe's customization -> RecipeMeta. THE only exec() of recipe_meta.py.
Missing file -> all registry defaults (the zero-config baseline, spec §2). Unknown
non-underscore ALL-CAPS top-level name or type mismatch -> MetaError (hard error).
`tests_dir` overrides the recipe-meta root (unit tests / fixtures)."""
path = meta_path(recipe, tests_dir)
values = {k.name: copy.copy(k.default) for k in KEYS}
if os.path.exists(path):
ns: dict = {}
with open(path) as fh:
exec(compile(fh.read(), path, "exec"), ns) # noqa: S102 (trusted, in-repo)
for name in sorted(ns):
if name.startswith("_") or not name.isupper():
continue # _FOO = recipe-private (exempt); lowercase = helpers/imports (ignored)
key = _REGISTRY.get(name)
if key is None:
near = difflib.get_close_matches(name, _REGISTRY, n=1)
hint = f" — did you mean {near[0]!r}?" if near else ""
raise MetaError(
f"{path}: unknown recipe_meta key {name!r}{hint}. Registered keys: "
f"{', '.join(sorted(_REGISTRY))}. Recipe-private constants must be "
f"underscore-prefixed (e.g. _{name})."
)
values[name] = _coerce(key, ns[name], path)
if key.hook_params and callable(values[name]):
check_hook_signature(values[name], key.hook_params, f"{path}: {name}")
if key.validate:
key.validate(values[name])
return RecipeMeta(**values)
def as_dict(meta) -> dict:
"""RecipeMeta -> {key: value} (every registered key, defaults included)."""
return dataclasses.asdict(meta)
def non_default(meta) -> dict:
"""The keys a recipe explicitly customized: {key: value} where value differs from the registry
default. Hooks compare by identity-vs-None (a set hook is always non-default). Feeds the run's
customization manifest (P5)."""
out = {}
for k in KEYS:
v = getattr(meta, k.name)
if v != k.default:
out[k.name] = v
return out
@dataclasses.dataclass(frozen=True)
class HookCtx:
"""The single argument every recipe hook receives (rcust P3 uniform ctx convention):
`EXTRA_ENV(ctx)`, `UPGRADE_EXTRA_ENV(ctx)`, `READY_PROBE(ctx)`, `BACKUP_VERIFY(ctx)`,
`SCREENSHOT(page, ctx)`, ops.py `pre_<op>(ctx)`."""
domain: str # the app's per-run domain
base_url: str # https://<domain>
meta: object # the recipe's full RecipeMeta
deps: dict | None # provisioned dep creds ({dep_recipe: entry}) or None if absent/empty
op: str | None # current lifecycle op (install|upgrade|backup|restore) or None
def _run_deps() -> dict | None:
"""The current run's provisioned dep creds from $CCCI_DEPS_FILE (either shape), or None.
Read directly (not via harness.deps) to keep meta.py import-cycle-free."""
path = os.environ.get("CCCI_DEPS_FILE")
if not path or not os.path.exists(path):
return None
try:
with open(path) as f:
data = json.load(f)
except (OSError, ValueError):
return None
if isinstance(data, dict):
return data or None
if isinstance(data, list):
out = {e["recipe"]: e for e in data if isinstance(e, dict) and e.get("recipe")}
return out or None
return None
def hook_ctx(domain: str, meta, *, op: str | None = None) -> HookCtx:
"""Build the HookCtx for a hook call site. Dep creds are picked up from the run's
$CCCI_DEPS_FILE when present (None otherwise)."""
return HookCtx(domain=domain, base_url=f"https://{domain}", meta=meta, deps=_run_deps(), op=op)
def _env_map(value, ctx: HookCtx) -> dict[str, str]:
if callable(value):
value = value(ctx)
return {str(k): str(v) for k, v in (value or {}).items()}
def extra_env(meta, ctx: HookCtx) -> dict[str, str]:
"""Resolve EXTRA_ENV (dict or callable(ctx)->dict) to the concrete per-run env map."""
return _env_map(meta.EXTRA_ENV, ctx)
def upgrade_extra_env(meta, ctx: HookCtx) -> dict[str, str]:
"""Resolve UPGRADE_EXTRA_ENV (dict or callable(ctx)->dict) to the concrete env map."""
return _env_map(meta.UPGRADE_EXTRA_ENV, ctx)

4
runner/harness/results.py
View File

@ -203,6 +203,7 @@ def build_results(
screenshot: str | None = None, screenshot: str | None = None,
summary_card: str | None = None, summary_card: str | None = None,
expected_na: dict | None = None, expected_na: dict | None = None,
customization: dict | None = None,
) -> dict: ) -> dict:
"""Assemble the full results.json dict (no I/O). `finished_ts` is passed in (the orchestrator """Assemble the full results.json dict (no I/O). `finished_ts` is passed in (the orchestrator
stamps it) so this stays pure and deterministic for unit tests. `expected_na` is the recipe's stamps it) so this stays pure and deterministic for unit tests. `expected_na` is the recipe's
@ -236,6 +237,9 @@ def build_results(
}, },
"screenshot": screenshot, "screenshot": screenshot,
"summary_card": summary_card, "summary_card": summary_card,
# rcust P5: the run's resolved customization manifest (pure presentation — consumers must
# never derive a verdict from it).
"customization": customization,
} }

23
runner/harness/screenshot.py
View File

@ -8,7 +8,7 @@ Secret-safety (R7, the cardinal screenshot guardrail): the screenshot step must
that displays generated credentials (an install wizard showing the initial admin password, a secrets that displays generated credentials (an install wizard showing the initial admin password, a secrets
page, etc.). The DEFAULT capture is the app's **landing page** (a login form shows fields, not the page, etc.). The DEFAULT capture is the app's **landing page** (a login form shows fields, not the
password) — safe for every recipe. A recipe that needs a post-login view opts in via a recipe-meta password) — safe for every recipe. A recipe that needs a post-login view opts in via a recipe-meta
`SCREENSHOT` hook: a callable `screenshot(page, domain, meta) -> None` that drives Playwright to a `SCREENSHOT` hook: a callable `SCREENSHOT(page, ctx) -> None` that drives Playwright to a
safe, credential-free view and is responsible for not landing on a secrets page. The harness never safe, credential-free view and is responsible for not landing on a secrets page. The harness never
auto-fills a wizard. auto-fills a wizard.
@ -21,6 +21,7 @@ from __future__ import annotations
import os import os
from . import browser as harness_browser from . import browser as harness_browser
from . import meta as meta_mod
# Default viewport for the captured screenshot — a desktop-ish frame that crops well into the card. # Default viewport for the captured screenshot — a desktop-ish frame that crops well into the card.
VIEWPORT = {"width": 1280, "height": 800} VIEWPORT = {"width": 1280, "height": 800}
@ -33,12 +34,19 @@ def screenshot_path(run_artifact_dir: str) -> str:
return os.path.join(run_artifact_dir, "screenshot.png") return os.path.join(run_artifact_dir, "screenshot.png")
def _load_screenshot_hook(recipe_meta: dict | None): def _load_screenshot_hook(recipe_meta):
"""Return the recipe's optional SCREENSHOT hook (a callable) if it declared one, else None. """Return the recipe's optional SCREENSHOT hook (a callable) if it declared one, else None.
The hook drives Playwright to a safe post-login view; default is the landing page.""" The hook drives Playwright to a safe post-login view; default is the landing page.
if not recipe_meta:
`recipe_meta` is the loaded RecipeMeta (rcust P1 — the single loader actually delivers
SCREENSHOT now; under the old L1 allowlist the key never arrived, spec §8 R2). A plain dict
is still accepted for direct/manual callers."""
if recipe_meta is None:
return None return None
hook = recipe_meta.get("SCREENSHOT") if isinstance(recipe_meta, dict):
hook = recipe_meta.get("SCREENSHOT")
else:
hook = getattr(recipe_meta, "SCREENSHOT", None)
return hook if callable(hook) else None return hook if callable(hook) else None
@ -67,8 +75,9 @@ def capture(domain: str, out_path: str, *, recipe_meta: dict | None = None) -> s
if hook is not None: if hook is not None:
# Recipe-specific safe view (post-login etc.). The hook owns navigation + # Recipe-specific safe view (post-login etc.). The hook owns navigation +
# the no-secret-page guarantee; it should call page.screenshot itself, but if # the no-secret-page guarantee; it should call page.screenshot itself, but if
# it doesn't, we still snap the resulting page below. # it doesn't, we still snap the resulting page below. SCREENSHOT(page, ctx) —
hook(page, domain, recipe_meta) # the uniform ctx convention (rcust P3).
hook(page, meta_mod.hook_ctx(domain, recipe_meta))
if not os.path.exists(out_path): if not os.path.exists(out_path):
page.screenshot(path=out_path, full_page=False) page.screenshot(path=out_path, full_page=False)
else: else:

270
runner/run_recipe_ci.py
View File

@ -58,6 +58,12 @@ from harness import ( # noqa: E402
from harness import ( # noqa: E402 from harness import ( # noqa: E402
deps as deps_mod, deps as deps_mod,
) )
from harness import ( # noqa: E402
manifest as manifest_mod,
)
from harness import ( # noqa: E402
meta as meta_mod,
)
from harness import ( # noqa: E402 from harness import ( # noqa: E402
results as results_mod, results as results_mod,
) )
@ -70,7 +76,7 @@ ALL_STAGES = ("install", "upgrade", "backup", "restore", "custom")
def sso_dep_unverified(declared, deps_ready: bool, requires_deps_skipped: int) -> bool: def sso_dep_unverified(declared, deps_ready: bool, requires_deps_skipped: int) -> bool:
"""F2-11 gate predicate (pure, unit-tested). True when a recipe declares DEPS but its """F2-11 gate predicate (pure, unit-tested). True when a recipe declares DEPS but its
setup_custom_tests failed (deps not ready) AND that caused ≥1 `requires_deps` (SSO/OIDC) test dep provisioning failed (deps not ready) AND that caused ≥1 `requires_deps` (SSO/OIDC) test
to SKIP. In that case the recipe's characteristic SSO claim was NOT verified, so the run must to SKIP. In that case the recipe's characteristic SSO claim was NOT verified, so the run must
NOT report GREEN — even though a skip-only pytest file exits 0 and leaves every tier 'pass'. NOT report GREEN — even though a skip-only pytest file exits 0 and leaves every tier 'pass'.
Generic-tier failure-isolation is preserved (those results stand); only the green SIGNAL is Generic-tier failure-isolation is preserved (those results stand); only the green SIGNAL is
@ -247,52 +253,29 @@ def snapshot_recipe_tests(recipe: str) -> str | None:
return dst return dst
def _load_meta(recipe: str) -> dict:
"""Mirror tests/conftest._recipe_meta so the orchestrator's deploy/wait uses the same per-recipe
config the tiers see (timeouts, health path/codes)."""
meta = {
"HEALTH_PATH": "/",
"HEALTH_OK": (200, 301, 302),
"DEPLOY_TIMEOUT": 600,
"HTTP_TIMEOUT": 300,
}
path = os.path.join(ROOT, "tests", recipe, "recipe_meta.py")
if os.path.exists(path):
ns: dict = {}
with open(path) as fh:
exec(compile(fh.read(), path, "exec"), ns) # noqa: S102 (trusted, in-repo)
for k in list(meta) + [
"BACKUP_CAPABLE",
"SKIP_GENERIC",
"EXPECTED_NA",
"OIDC_AT_INSTALL",
"READY_PROBE",
"UPGRADE_BASE_VERSION",
"BACKUP_VERIFY",
"UPGRADE_EXTRA_ENV",
]:
if k in ns:
meta[k] = ns[k]
return meta
def _tier_env(domain: str) -> dict: def _tier_env(domain: str) -> dict:
return dict(os.environ, CCCI_APP_DOMAIN=domain, CCCI_BASE_URL=f"https://{domain}") return dict(os.environ, CCCI_APP_DOMAIN=domain, CCCI_BASE_URL=f"https://{domain}")
def _skip_generic(op: str, meta: dict) -> bool: def skip_generic_env_overrides() -> list[str]:
"""Active CCCI_SKIP_GENERIC* env overrides (rcust P2c: the meta key is deleted; the env form
is a documented LOCAL-DEV-ONLY escape hatch). Surfaced loudly when set in a CI (drone) run —
it reduces generic-floor coverage and must never silently ride a CI verdict."""
return sorted(
k for k in os.environ if k.startswith("CCCI_SKIP_GENERIC") and _truthy(os.environ.get(k))
)
def _skip_generic(op: str) -> bool:
"""Whether the generic assertion for `op` is opted out (Phase 1e HC3). Default: run (additive). """Whether the generic assertion for `op` is opted out (Phase 1e HC3). Default: run (additive).
Opt-out, any of: env CCCI_SKIP_GENERIC (all ops), env CCCI_SKIP_GENERIC_<OP>, or the recipe's Opt-out via env only (dev-only escape hatch, P2c): CCCI_SKIP_GENERIC (all ops) or
declarative recipe_meta.SKIP_GENERIC list (op name, or "all"/"*").""" CCCI_SKIP_GENERIC_<OP>. The recipe_meta SKIP_GENERIC key is deleted (zero users)."""
if _truthy(os.environ.get("CCCI_SKIP_GENERIC")): if _truthy(os.environ.get("CCCI_SKIP_GENERIC")):
return True return True
if _truthy(os.environ.get(f"CCCI_SKIP_GENERIC_{op.upper()}")): return _truthy(os.environ.get(f"CCCI_SKIP_GENERIC_{op.upper()}"))
return True
sg = [str(s).lower() for s in (meta.get("SKIP_GENERIC") or [])]
return "all" in sg or "*" in sg or op in sg
def _run_pre_hook(recipe: str, op: str, repo_local: str | None, domain: str, meta: dict) -> None: def _run_pre_hook(recipe: str, op: str, repo_local: str | None, domain: str, meta) -> None:
"""Run the optional pre-op seed hook (recipe ops.py `pre_<op>`) BEFORE the harness performs the """Run the optional pre-op seed hook (recipe ops.py `pre_<op>`) BEFORE the harness performs the
op (HC3 op/assertion split): overlays seed data-continuity markers / the backup→restore mutation op (HC3 op/assertion split): overlays seed data-continuity markers / the backup→restore mutation
here, then assert post-op in test_<op>.py. cc-ci's ops.py is trusted; a repo-local ops.py is here, then assert post-op in test_<op>.py. cc-ci's ops.py is trusted; a repo-local ops.py is
@ -309,7 +292,11 @@ def _run_pre_hook(recipe: str, op: str, repo_local: str | None, domain: str, met
mod = importlib.util.module_from_spec(spec) mod = importlib.util.module_from_spec(spec)
spec.loader.exec_module(mod) spec.loader.exec_module(mod)
print(f" pre-op seed ({source}): {os.path.relpath(path, ROOT)}::pre_{op}", flush=True) print(f" pre-op seed ({source}): {os.path.relpath(path, ROOT)}::pre_{op}", flush=True)
getattr(mod, f"pre_{op}")(domain, meta) fn = getattr(mod, f"pre_{op}")
# Uniform ctx convention (rcust P3): pre_<op>(ctx). A legacy (domain, meta) hook fails
# HERE with a clear migration message, not a TypeError mid-call.
meta_mod.check_hook_signature(fn, ("ctx",), f"{os.path.relpath(path, ROOT)}::pre_{op}")
fn(meta_mod.hook_ctx(domain, meta, op=op))
finally: finally:
if d in sys.path: if d in sys.path:
sys.path.remove(d) sys.path.remove(d)
@ -322,7 +309,7 @@ def _perform_op(
head_ref: str | None, head_ref: str | None,
op_state: dict, op_state: dict,
deploy_timeout: int = 900, deploy_timeout: int = 900,
meta: dict | None = None, meta=None,
) -> None: ) -> None:
"""Perform the single mutating op ONCE (the harness owns the op, HC3). install has no op. Records """Perform the single mutating op ONCE (the harness owns the op, HC3). install has no op. Records
what the assertions need (pre-upgrade identity, backup snapshot_id) into op_state. None of these what the assertions need (pre-upgrade identity, backup snapshot_id) into op_state. None of these
@ -345,9 +332,10 @@ def _perform_op(
# verify fails we re-run the WHOLE backup (fresh restic snapshot) with a re-stabilised DB, up to # verify fails we re-run the WHOLE backup (fresh restic snapshot) with a re-stabilised DB, up to
# 3 attempts. Recipes without BACKUP_VERIFY are unaffected (single backup, as before). # 3 attempts. Recipes without BACKUP_VERIFY are unaffected (single backup, as before).
snap = generic.perform_backup(domain) snap = generic.perform_backup(domain)
verify = meta.get("BACKUP_VERIFY") if meta else None verify = meta.BACKUP_VERIFY if meta else None
verify_ctx = meta_mod.hook_ctx(domain, meta, op="backup") if meta else None
attempt = 1 attempt = 1
while callable(verify) and not verify(domain) and attempt < 3: while callable(verify) and not verify(verify_ctx) and attempt < 3:
attempt += 1 attempt += 1
print( print(
f" backup-verify FAILED (attempt {attempt - 1}/3) — backup did not capture the " f" backup-verify FAILED (attempt {attempt - 1}/3) — backup did not capture the "
@ -355,7 +343,7 @@ def _perform_op(
flush=True, flush=True,
) )
snap = generic.perform_backup(domain) snap = generic.perform_backup(domain)
if callable(verify) and not verify(domain): if callable(verify) and not verify(verify_ctx):
print( print(
f" !! backup-verify still FAILED after {attempt} attempts — backup is incomplete", f" !! backup-verify still FAILED after {attempt} attempts — backup is incomplete",
flush=True, flush=True,
@ -371,7 +359,7 @@ def run_lifecycle_tier(
op: str, op: str,
repo_local: str | None, repo_local: str | None,
domain: str, domain: str,
meta: dict, meta,
head_ref: str | None, head_ref: str | None,
op_state: dict, op_state: dict,
records: list[dict] | None = None, records: list[dict] | None = None,
@ -386,7 +374,7 @@ def run_lifecycle_tier(
a {tier,source,file,rc,junit} record appended, so the run can assemble per-stage/per-test a {tier,source,file,rc,junit} record appended, so the run can assemble per-stage/per-test
results.json + the level afterwards. Purely additive — does not change the verdict.""" results.json + the level afterwards. Purely additive — does not change the verdict."""
overlay = discovery.resolve_overlay_op(recipe, op, repo_local) overlay = discovery.resolve_overlay_op(recipe, op, repo_local)
skip_gen = _skip_generic(op, meta) skip_gen = _skip_generic(op)
files: list[tuple[str, str]] = [] files: list[tuple[str, str]] = []
if not skip_gen: if not skip_gen:
files.append(discovery.generic_op(op)) files.append(discovery.generic_op(op))
@ -411,7 +399,7 @@ def run_lifecycle_tier(
recipe, recipe,
head_ref, head_ref,
op_state, op_state,
deploy_timeout=int(meta.get("DEPLOY_TIMEOUT", 900)), deploy_timeout=int(meta.DEPLOY_TIMEOUT),
meta=meta, meta=meta,
) )
with open(os.environ["CCCI_OP_STATE_FILE"], "w") as f: with open(os.environ["CCCI_OP_STATE_FILE"], "w") as f:
@ -449,7 +437,7 @@ def run_lifecycle_tier(
def _enrich_deps_with_sso(parent_recipe: str, parent_domain: str, deps_list) -> dict[str, dict]: def _enrich_deps_with_sso(parent_recipe: str, parent_domain: str, deps_list) -> dict[str, dict]:
"""For each dep, set up a fresh realm/client + test user via the harness's provider-specific """For each dep, set up a fresh realm/client + test user via the harness's provider-specific
setup function, then return a recipe→entry dict carrying domain + admin + realm/client/user setup function, then return a recipe→entry dict carrying domain + admin + realm/client/user
info — the shape the `setup_custom_tests.sh` hook (and dependent tests) read. info — the shape the `install_steps.sh` hook (and dependent tests) read.
Provider routing: today only `keycloak` is supported. authentik will need a parallel Provider routing: today only `keycloak` is supported. authentik will need a parallel
`setup_authentik_realm` when an authentik-dep recipe enrolls (DEFERRED.md #9). `setup_authentik_realm` when an authentik-dep recipe enrolls (DEFERRED.md #9).
@ -463,7 +451,7 @@ def _enrich_deps_with_sso(parent_recipe: str, parent_domain: str, deps_list) ->
if not dep_recipe or not dep_domain: if not dep_recipe or not dep_domain:
continue continue
if dep_recipe != "keycloak": if dep_recipe != "keycloak":
# Provider not yet supported — record bare entry; setup_custom_tests.sh / tests will # Provider not yet supported — record bare entry; install_steps.sh / tests will
# raise if they need realm/client info they don't see. # raise if they need realm/client info they don't see.
out[dep_recipe] = entry out[dep_recipe] = entry
continue continue
@ -507,12 +495,10 @@ def _provision_deps(
Splits deps into live-warm (shared provider at a stable domain + a per-run realm) vs cold Splits deps into live-warm (shared provider at a stable domain + a per-run realm) vs cold
(co-deployed per run), provisions each dep's SSO realm/client/user, and persists the enriched (co-deployed per run), provisions each dep's SSO realm/client/user, and persists the enriched
dict the `setup_custom_tests.sh`/`install_steps.sh` hooks + dependent tests read. Raises on any dict the `install_steps.sh` hooks + dependent tests read. Raises on any failure (the caller
failure (the caller marks deps-not-ready). Used by BOTH wiring paths: marks deps-not-ready). Install-time wiring is the ONLY mode (rcust P2b): provision BEFORE the
- post-deploy (legacy): provision AFTER generic tiers, then `setup_custom_tests.sh` does an single deploy so the install-tier `install_steps.sh` hook wires OIDC env into that one deploy —
in-place OIDC redeploy. no reconverge, no post-deploy `setup_custom_tests.sh` machinery.
- install-time (`OIDC_AT_INSTALL`, Q3.2a): provision BEFORE the single deploy so the
install-tier `install_steps.sh` hook wires OIDC env into that one deploy — no reconverge.
""" """
warm_deps, cold_deps = [], [] warm_deps, cold_deps = [], []
for d in declared: for d in declared:
@ -523,7 +509,7 @@ def _provision_deps(
if wd: if wd:
print(f" dep: {d} warm provider {wd} not up — cold fallback", flush=True) print(f" dep: {d} warm provider {wd} not up — cold fallback", flush=True)
cold_deps.append(d) cold_deps.append(d)
dep_metas = {d: _load_meta(d) for d in cold_deps} dep_metas = {d: meta_mod.load(d) for d in cold_deps}
deps_list = ( deps_list = (
deps_mod.deploy_deps(recipe, os.environ.get("PR", "0"), ref, cold_deps, meta_for=dep_metas) deps_mod.deploy_deps(recipe, os.environ.get("PR", "0"), ref, cold_deps, meta_for=dep_metas)
if cold_deps if cold_deps
@ -541,32 +527,6 @@ def _provision_deps(
return deps_state return deps_state
def _run_setup_custom_tests_hook(recipe: str, domain: str, deps_file: str) -> None:
"""Run `tests/<recipe>/setup_custom_tests.sh` if present (operator-2026-05-28 SSO-dep plan
§3.2). The hook reads `$CCCI_DEPS_FILE`, sets OIDC env via `abra app config set` + secret
insert, and triggers an in-place `abra app deploy --force --chaos`. Failure here propagates
to mark deps-not-ready (caught in main())."""
path = os.path.join(ROOT, "tests", recipe, "setup_custom_tests.sh")
if not os.path.isfile(path):
# No hook = recipe doesn't need post-deps wiring; deps are deployed + creds available
# via deps_apps fixture as-is.
print(
f" setup_custom_tests: no hook at {os.path.relpath(path, ROOT)} (deps creds ready in $CCCI_DEPS_FILE)",
flush=True,
)
return
print(f" setup_custom_tests hook: {os.path.relpath(path, ROOT)}", flush=True)
rc = subprocess.run(
["bash", path],
check=False,
env=dict(os.environ, CCCI_APP_DOMAIN=domain, CCCI_RECIPE=recipe, CCCI_DEPS_FILE=deps_file),
)
if rc.returncode != 0:
raise RuntimeError(
f"setup_custom_tests.sh exited {rc.returncode} (deps env not wired into parent)"
)
def run_custom( def run_custom(
recipe: str, recipe: str,
repo_local: str | None, repo_local: str | None,
@ -609,7 +569,7 @@ def _wait_undeployed(domain: str, timeout: int = 120) -> None:
def run_quick( def run_quick(
recipe: str, ref: str | None, head_ref: str | None, repo_local: str | None, meta: dict recipe: str, ref: str | None, head_ref: str | None, repo_local: str | None, meta
) -> int: ) -> int:
"""WC4 `--quick` opt-in fast lane (plan §2). Reattach the data-warm canonical (known-good volume) """WC4 `--quick` opt-in fast lane (plan §2). Reattach the data-warm canonical (known-good volume)
→ upgrade IN PLACE to the PR head (chaos) → assert generic UPGRADE (reconverge+moved+serving) + → upgrade IN PLACE to the PR head (chaos) → assert generic UPGRADE (reconverge+moved+serving) +
@ -645,7 +605,7 @@ def run_quick(
op_state: dict = {} op_state: dict = {}
results: dict[str, str] = {} results: dict[str, str] = {}
declared = deps_mod.declared_deps(recipe) declared = list(meta.DEPS)
deps_state: dict = {} deps_state: dict = {}
deps_ready = True deps_ready = True
deps_not_ready_reason = "" deps_not_ready_reason = ""
@ -657,28 +617,32 @@ def run_quick(
try: try:
# 1) reattach the canonical (warm boot at the known-good version + retained volume) # 1) reattach the canonical (warm boot at the known-good version + retained volume)
try: try:
canonical.deploy_canonical(recipe, timeout=int(meta.get("DEPLOY_TIMEOUT", 900))) canonical.deploy_canonical(recipe, timeout=int(meta.DEPLOY_TIMEOUT))
lifecycle.wait_healthy( lifecycle.wait_healthy(
domain, domain,
ok_codes=tuple(meta["HEALTH_OK"]), ok_codes=tuple(meta.HEALTH_OK),
path=meta["HEALTH_PATH"], path=meta.HEALTH_PATH,
deploy_timeout=meta["DEPLOY_TIMEOUT"], deploy_timeout=meta.DEPLOY_TIMEOUT,
http_timeout=meta["HTTP_TIMEOUT"], http_timeout=meta.HTTP_TIMEOUT,
) )
warm_ok = True warm_ok = True
except Exception as e: # noqa: BLE001 except Exception as e: # noqa: BLE001
print(f"!! canonical reattach/readiness failed: {_scrub(str(e))}", flush=True) print(f"!! canonical reattach/readiness failed: {_scrub(str(e))}", flush=True)
if warm_ok: if warm_ok:
# 2) deps (warm keycloak + per-run realm) — mirrors main()'s warm/cold split # 2) deps (warm keycloak + per-run realm) — mirrors main()'s warm/cold split. NB
# (rcust P2b): deps are provisioned (realm/creds in $CCCI_DEPS_FILE) but quick mode
# cannot do install-time OIDC env wiring — the canonical app pre-exists its per-run
# realm. No quick-enrolled recipe declares DEPS today; if one ever does, its
# requires_deps tests will exercise creds-only flows or skip (F2-11 keeps the signal).
if declared: if declared:
print(f"\n===== setup_custom_tests (quick): deps {declared} =====", flush=True) print(f"\n===== deps (quick): {declared} =====", flush=True)
try: try:
warm_deps, cold_deps = [], [] warm_deps, cold_deps = [], []
for d in declared: for d in declared:
wd = warm.warm_domain(d) wd = warm.warm_domain(d)
(warm_deps if (wd and warm.is_warm_up(d, wd)) else cold_deps).append(d) (warm_deps if (wd and warm.is_warm_up(d, wd)) else cold_deps).append(d)
dep_metas = {d: _load_meta(d) for d in cold_deps} dep_metas = {d: meta_mod.load(d) for d in cold_deps}
deps_list = ( deps_list = (
deps_mod.deploy_deps( deps_mod.deploy_deps(
recipe, os.environ.get("PR", "0"), ref, cold_deps, meta_for=dep_metas recipe, os.environ.get("PR", "0"), ref, cold_deps, meta_for=dep_metas
@ -693,12 +657,11 @@ def run_quick(
print(f" dep: using live-warm {d} @ {wd} (per-run realm)", flush=True) print(f" dep: using live-warm {d} @ {wd} (per-run realm)", flush=True)
deps_state = _enrich_deps_with_sso(recipe, domain, deps_list) deps_state = _enrich_deps_with_sso(recipe, domain, deps_list)
deps_mod.write_run_state(deps_state) deps_mod.write_run_state(deps_state)
_run_setup_custom_tests_hook(recipe, domain, depsfile)
except Exception as e: # noqa: BLE001 except Exception as e: # noqa: BLE001
deps_ready = False deps_ready = False
deps_not_ready_reason = _scrub(str(e))[:300] deps_not_ready_reason = _scrub(str(e))[:300]
print( print(
f"!! setup_custom_tests failed (deps-not-ready): {deps_not_ready_reason}", f"!! dep provisioning failed (deps-not-ready): {deps_not_ready_reason}",
flush=True, flush=True,
) )
@ -813,7 +776,7 @@ def run_quick(
overall = 1 overall = 1
if sso_unverified: if sso_unverified:
print( print(
f"!! DEPS={declared} but setup_custom_tests failed and {requires_deps_skipped} " f"!! DEPS={declared} but dep provisioning failed and {requires_deps_skipped} "
"requires_deps SKIPPED — SSO NOT verified (F2-11)", "requires_deps SKIPPED — SSO NOT verified (F2-11)",
file=sys.stderr, file=sys.stderr,
) )
@ -848,7 +811,7 @@ def promote_canonical(recipe: str, head_ref: str | None) -> None:
if not latest: if not latest:
print(f"WC5 promote: no version tags for {recipe} — skip", flush=True) print(f"WC5 promote: no version tags for {recipe} — skip", flush=True)
return return
meta = _load_meta(recipe) meta = meta_mod.load(recipe)
# The cold run's deploy-count was already asserted + the countfile removed; don't perturb it. # The cold run's deploy-count was already asserted + the countfile removed; don't perturb it.
os.environ.pop("CCCI_DEPLOY_COUNT_FILE", None) os.environ.pop("CCCI_DEPLOY_COUNT_FILE", None)
print( print(
@ -860,14 +823,15 @@ def promote_canonical(recipe: str, head_ref: str | None) -> None:
domain, domain,
version=latest, version=latest,
secrets=True, secrets=True,
deploy_timeout=int(meta.get("DEPLOY_TIMEOUT", 900)), deploy_timeout=int(meta.DEPLOY_TIMEOUT),
meta=meta,
) )
lifecycle.wait_healthy( lifecycle.wait_healthy(
domain, domain,
ok_codes=tuple(meta["HEALTH_OK"]), ok_codes=tuple(meta.HEALTH_OK),
path=meta["HEALTH_PATH"], path=meta.HEALTH_PATH,
deploy_timeout=meta["DEPLOY_TIMEOUT"], deploy_timeout=meta.DEPLOY_TIMEOUT,
http_timeout=meta["HTTP_TIMEOUT"], http_timeout=meta.HTTP_TIMEOUT,
) )
abra.undeploy(domain) abra.undeploy(domain)
_wait_undeployed(domain) _wait_undeployed(domain)
@ -896,6 +860,17 @@ def main() -> int:
print( print(
f"== cc-ci run: recipe={recipe} ref={ref} pr={os.environ.get('PR', '0')} stages={sorted(stages)}" f"== cc-ci run: recipe={recipe} ref={ref} pr={os.environ.get('PR', '0')} stages={sorted(stages)}"
) )
# P2c: the CCCI_SKIP_GENERIC* env escape hatch is LOCAL-DEV-ONLY. If it rides a CI (drone)
# run, shout — generic-floor coverage is reduced and the verdict must not look routine.
for ov in skip_generic_env_overrides():
if os.environ.get("DRONE"):
print(
f"!! {ov}=1 — dev-only generic-floor override ACTIVE IN A CI RUN; generic "
"assertions are suppressed for the affected op(s). This must never gate a merge.",
flush=True,
)
else:
print(f"== {ov}=1 (dev-only generic-floor override active)", flush=True)
# Concurrent-run safety is structural: this run's recipe trees live in its own ABRA_DIR # Concurrent-run safety is structural: this run's recipe trees live in its own ABRA_DIR
# (exported here, before ANY abra call), so no recipe-tree lock exists; same-DOMAIN runs # (exported here, before ANY abra call), so no recipe-tree lock exists; same-DOMAIN runs
# serialise on the app-domain flock taken in deploy_app (see docs/concurrency.md). # serialise on the app-domain flock taken in deploy_app (see docs/concurrency.md).
@ -906,7 +881,13 @@ def main() -> int:
# HEAD (the catalogue current) for a non-PR `!testme`. Captured before any version-tag checkout. # HEAD (the catalogue current) for a non-PR `!testme`. Captured before any version-tag checkout.
head_ref = ref or lifecycle.recipe_head_commit(recipe) head_ref = ref or lifecycle.recipe_head_commit(recipe)
repo_local = snapshot_recipe_tests(recipe) repo_local = snapshot_recipe_tests(recipe)
meta = _load_meta(recipe) meta = meta_mod.load(recipe)
# Customization manifest (rcust P5, R4): ONE block answering "what does this recipe
# customize?" across all surfaces — printed here and embedded verbatim in results.json under
# "customization". Pure presentation; never influences a verdict.
customization = manifest_mod.build(recipe, meta, repo_local)
print("\n" + manifest_mod.render(recipe, customization) + "\n", flush=True)
# WC4/WC7: opt-in `--quick` fast lane. Requires an existing data-warm canonical; if none, fall # WC4/WC7: opt-in `--quick` fast lane. Requires an existing data-warm canonical; if none, fall
# back cleanly to the full COLD run below so the PR is still tested (DECISIONS Phase-2w). # back cleanly to the full COLD run below so the PR is still tested (DECISIONS Phase-2w).
@ -929,9 +910,7 @@ def main() -> int:
# override must be an exact published version tag (deployed as a pinned base). (Adversary §7.1.) # override must be an exact published version tag (deployed as a pinned base). (Adversary §7.1.)
want_upgrade = "upgrade" in stages want_upgrade = "upgrade" in stages
prev = ( prev = (
(meta.get("UPGRADE_BASE_VERSION") or lifecycle.previous_version(recipe)) (meta.UPGRADE_BASE_VERSION or lifecycle.previous_version(recipe)) if want_upgrade else None
if want_upgrade
else None
) )
base = prev or target base = prev or target
backup_cap = generic.backup_capable(recipe, meta) backup_cap = generic.backup_capable(recipe, meta)
@ -960,10 +939,8 @@ def main() -> int:
os.environ["CCCI_OP_STATE_FILE"] = statefile os.environ["CCCI_OP_STATE_FILE"] = statefile
op_state: dict = {} op_state: dict = {}
# Run-scoped dep state (Phase 2 Q2.3, refined per operator-2026-05-28 SSO-dep plan §1): # Run-scoped dep state (Phase 2 Q2.3; install-time-only since rcust P2b): deps are provisioned
# deps now deploy AFTER generic tiers (between RESTORE and CUSTOM) so a failed dep deploy # BEFORE the single deploy so install_steps.sh wires OIDC env into that one deploy.
# cannot break the generic-tier signal. The `setup_custom_tests` step deploys each dep + runs
# `tests/<recipe>/setup_custom_tests.sh` to wire OIDC env via in-place redeploy.
# `$CCCI_DEPS_FILE` is written with the full creds dict the hook script needs (jq-readable). # `$CCCI_DEPS_FILE` is written with the full creds dict the hook script needs (jq-readable).
depsfile = _run_state_path("deps") + ".json" depsfile = _run_state_path("deps") + ".json"
with open(depsfile, "w") as f: with open(depsfile, "w") as f:
@ -974,15 +951,9 @@ def main() -> int:
with contextlib.suppress(OSError): with contextlib.suppress(OSError):
os.remove(skipfile) os.remove(skipfile)
os.environ["CCCI_DEPS_SKIP_REPORT"] = skipfile os.environ["CCCI_DEPS_SKIP_REPORT"] = skipfile
declared = deps_mod.declared_deps(recipe) declared = list(meta.DEPS)
# Q3.2a: a recipe that tolerates OIDC env at first boot AND whose deps are live-warm wires OIDC
# at INSTALL time (provision the realm BEFORE the single deploy; install_steps.sh writes the env
# into it) instead of the post-deploy in-place `--chaos` redeploy — which is flaky on the heavy
# 12-service lasuite-drive stack (collabora WOPI race; see JOURNAL Step 0). Opt-in per recipe.
oidc_at_install = bool(meta.get("OIDC_AT_INSTALL")) and bool(declared)
if declared: if declared:
when = "BEFORE deploy (install-time OIDC)" if oidc_at_install else "AFTER generic tiers" print(f"\n===== DEPS declared (provision BEFORE deploy): {declared} =====", flush=True)
print(f"\n===== DEPS declared (provision {when}): {declared} =====", flush=True)
deps_state: dict[str, dict] = {} # new shape: recipe→entry dict (sso-dep plan §1) deps_state: dict[str, dict] = {} # new shape: recipe→entry dict (sso-dep plan §1)
deps_ready = True deps_ready = True
deps_not_ready_reason: str = "" deps_not_ready_reason: str = ""
@ -996,7 +967,7 @@ def main() -> int:
# install_steps.sh can read $CCCI_DEPS_FILE and wire the OIDC env into that one deploy. On # install_steps.sh can read $CCCI_DEPS_FILE and wire the OIDC env into that one deploy. On
# failure we mark deps-not-ready but STILL deploy the recipe alone (install_steps.sh no-ops # failure we mark deps-not-ready but STILL deploy the recipe alone (install_steps.sh no-ops
# on an empty deps file) so the generic tiers run; the OIDC custom test then skips → F2-11. ---- # on an empty deps file) so the generic tiers run; the OIDC custom test then skips → F2-11. ----
if oidc_at_install: if declared:
print( print(
f"\n===== install-time OIDC: provisioning deps {declared} BEFORE deploy =====", f"\n===== install-time OIDC: provisioning deps {declared} BEFORE deploy =====",
flush=True, flush=True,
@ -1023,18 +994,21 @@ def main() -> int:
version=base, version=base,
secrets=True, secrets=True,
install_steps_hook=hook, install_steps_hook=hook,
deploy_timeout=int(meta.get("DEPLOY_TIMEOUT", 900)), deploy_timeout=int(meta.DEPLOY_TIMEOUT),
meta=meta,
) )
lifecycle.wait_healthy( lifecycle.wait_healthy(
domain, domain,
ok_codes=tuple(meta["HEALTH_OK"]), ok_codes=tuple(meta.HEALTH_OK),
path=meta["HEALTH_PATH"], path=meta.HEALTH_PATH,
deploy_timeout=meta["DEPLOY_TIMEOUT"], deploy_timeout=meta.DEPLOY_TIMEOUT,
http_timeout=meta["HTTP_TIMEOUT"], http_timeout=meta.HTTP_TIMEOUT,
) )
# Recipe READY_PROBE (e.g. lasuite-drive collabora WOPI discovery) — readiness beyond # Recipe READY_PROBE (e.g. lasuite-drive collabora WOPI discovery) — readiness beyond
# replica convergence + app HEALTH_PATH; no-op for recipes without one. # replica convergence + app HEALTH_PATH; no-op for recipes without one.
lifecycle.wait_ready_probes(meta, domain, timeout=int(meta.get("DEPLOY_TIMEOUT", 900))) lifecycle.wait_ready_probes(
meta, domain, timeout=int(meta.DEPLOY_TIMEOUT), op="install"
)
deploy_ok = True deploy_ok = True
except Exception as e: # noqa: BLE001 — a failed deploy is a reported INSTALL failure except Exception as e: # noqa: BLE001 — a failed deploy is a reported INSTALL failure
print(f"!! deploy/readiness failed: {e}", flush=True) print(f"!! deploy/readiness failed: {e}", flush=True)
@ -1131,41 +1105,11 @@ def main() -> int:
if backup_cap if backup_cap
else "skip" else "skip"
) )
# ---- setup_custom_tests step (NEW, operator-2026-05-28 SSO-dep plan §3.2) ---- # (rcust P2b: install-time deps wiring is the ONLY mode — deps were provisioned BEFORE
# Deploy each declared dep + wire OIDC env into the parent app via the per-recipe # the single deploy and install_steps.sh wired the OIDC env into it. The legacy
# setup_custom_tests.sh hook + in-place redeploy. Failure here marks deps-not-ready # post-deploy provisioning + setup_custom_tests.sh redeploy machinery is deleted; a
# but does NOT abort the run — @pytest.mark.requires_deps tests skip with reason; # recipe's post-deploy seeding belongs in ops.py pre_install, e.g. lasuite-drive's
# non-deps custom tests still run normally. # MinIO bucket one-shot.)
if declared and not oidc_at_install:
# LEGACY post-deploy path: provision deps AFTER generic tiers, then wire OIDC env
# into the parent via the setup_custom_tests.sh hook + an in-place `--chaos` redeploy.
print("\n===== setup_custom_tests: deps + OIDC wiring =====", flush=True)
try:
deps_state = _provision_deps(recipe, domain, ref, declared)
# Run the per-recipe post-deps hook (jq-driven OIDC wiring + in-place redeploy)
_run_setup_custom_tests_hook(recipe, domain, depsfile)
except Exception as e: # noqa: BLE001 — setup failure is ISOLATED to dep-marked tests
deps_ready = False
deps_not_ready_reason = _scrub(str(e))[:300]
print(
f"!! setup_custom_tests failed (deps-not-ready): {deps_not_ready_reason}",
flush=True,
)
elif declared and oidc_at_install and deps_ready:
# INSTALL-TIME path (Q3.2a): deps were provisioned BEFORE the single deploy and the
# install-tier install_steps.sh hook already wired OIDC env into that one deploy —
# so NO re-provision, NO reconverge here. Run only the post-deploy setup hook
# (e.g. lasuite-drive's minio-createbuckets one-shot), which needs the live stack.
print("\n===== post-deploy setup (OIDC already wired at install) =====", flush=True)
try:
_run_setup_custom_tests_hook(recipe, domain, depsfile)
except Exception as e: # noqa: BLE001 — isolated to dep-marked / state-dependent tests
deps_ready = False
deps_not_ready_reason = _scrub(str(e))[:300]
print(
f"!! post-deploy setup failed: {deps_not_ready_reason}",
flush=True,
)
# ---- CUSTOM tier ---- # ---- CUSTOM tier ----
if "custom" in stages: if "custom" in stages:
@ -1240,8 +1184,7 @@ def main() -> int:
# ---- per-op summary (DG6 feed) ---- # ---- per-op summary (DG6 feed) ----
# SSO-dep plan §1: DG4.1 generalised — one `abra app new` per app in the run (recipe + each # SSO-dep plan §1: DG4.1 generalised — one `abra app new` per app in the run (recipe + each
# COLD dep). In-place reconfigure-and-redeploy (the setup_custom_tests step's # COLD dep). Chaos redeploys are NOT a fresh `app_new` and do NOT increment the count.
# `abra app deploy --force --chaos`) is NOT a fresh `app_new` and does NOT increment the count.
# WC1: a live-warm dep (keycloak) is NOT deployed by the run — it only gets a per-run realm — so # WC1: a live-warm dep (keycloak) is NOT deployed by the run — it only gets a per-run realm — so
# warm deps contribute 0. So expected = 1 + (number of COLD deps that actually got deployed). # warm deps contribute 0. So expected = 1 + (number of COLD deps that actually got deployed).
_dep_entries = deps_state.values() if isinstance(deps_state, dict) else (deps_state or []) _dep_entries = deps_state.values() if isinstance(deps_state, dict) else (deps_state or [])
@ -1282,12 +1225,12 @@ def main() -> int:
overall = 1 overall = 1
if any(v == "fail" for v in results.values()): if any(v == "fail" for v in results.values()):
overall = 1 overall = 1
# F2-11: a deps-declaring recipe whose setup_custom_tests failed has NOT verified its SSO/OIDC # F2-11: a deps-declaring recipe whose dep provisioning failed has NOT verified its SSO/OIDC
# claim — its requires_deps tests SKIPPED (a skip-only file exits 0, so without this the run # claim — its requires_deps tests SKIPPED (a skip-only file exits 0, so without this the run
# would report GREEN). Fail the run for that recipe; generic-tier results above are untouched. # would report GREEN). Fail the run for that recipe; generic-tier results above are untouched.
if sso_dep_unverified(declared, deps_ready, requires_deps_skipped): if sso_dep_unverified(declared, deps_ready, requires_deps_skipped):
print( print(
f"!! recipe declares DEPS={declared} but setup_custom_tests failed and " f"!! recipe declares DEPS={declared} but dep provisioning failed and "
f"{requires_deps_skipped} requires_deps (SSO) test(s) were SKIPPED — SSO claim NOT " f"{requires_deps_skipped} requires_deps (SSO) test(s) were SKIPPED — SSO claim NOT "
f"verified; failing run (F2-11). deps-not-ready: {deps_not_ready_reason}", f"verified; failing run (F2-11). deps-not-ready: {deps_not_ready_reason}",
file=sys.stderr, file=sys.stderr,
@ -1314,7 +1257,8 @@ def main() -> int:
no_secret_leak=True, # narrowed below by an actual scan of the serialised artifact no_secret_leak=True, # narrowed below by an actual scan of the serialised artifact
screenshot=screenshot_rel, # Phase 3 U1 (R4): relative PNG name iff capture succeeded screenshot=screenshot_rel, # Phase 3 U1 (R4): relative PNG name iff capture succeeded
finished_ts=time.time(), finished_ts=time.time(),
expected_na=meta.get("EXPECTED_NA"), # declared intentional-skip map (recipe_meta) expected_na=meta.EXPECTED_NA, # declared intentional-skip map (recipe_meta)
customization=customization, # rcust P5: the run-start manifest, verbatim
) )
# Real (if narrow) leak check: no known infra-secret value may appear in the artifact (R7). # Real (if narrow) leak check: no known infra-secret value may appear in the artifact (R7).
blob = json.dumps(data) blob = json.dumps(data)

71
scripts/gen-meta-docs.py Normal file
View File

@ -0,0 +1,71 @@
#!/usr/bin/env python3
"""Render the harness.meta KEYS registry to the markdown key-reference table in
docs/recipe-customization.md §4 (rcust P1.5; kills the R5 doc-drift class).
Usage:
python3 scripts/gen-meta-docs.py # rewrite the table in-place between the markers
python3 scripts/gen-meta-docs.py --print # print the rendered table to stdout (used by the
# doc-sync unit test, tests/unit/test_meta.py)
The table lives between `<!-- META-TABLE-START -->` / `<!-- META-TABLE-END -->` markers; a unit
test asserts the committed table equals this rendering, so editing it by hand fails CI.
"""
from __future__ import annotations
import os
import sys
ROOT = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
sys.path.insert(0, os.path.join(ROOT, "runner"))
from harness.meta import KEYS # noqa: E402
DOC = os.path.join(ROOT, "docs", "recipe-customization.md")
START = "<!-- META-TABLE-START -->"
END = "<!-- META-TABLE-END -->"
def _default_repr(v) -> str:
if v is None:
return "`None`"
return f"`{v!r}`"
def render() -> str:
lines = [
START,
"",
"_This table is GENERATED from the `runner/harness/meta.py` KEYS registry by"
" `scripts/gen-meta-docs.py` — do not edit by hand (a unit test pins the sync)._",
"",
"| Key | Type | Default | Meaning |",
"|---|---|---|---|",
]
for k in KEYS:
doc = k.doc.replace("|", "\\|")
name = f"`{k.name}`" + (" **(deprecated)**" if k.deprecated else "")
lines.append(f"| {name} | `{k.type}` | {_default_repr(k.default)} | {doc} |")
lines += ["", END]
return "\n".join(lines)
def main() -> int:
table = render()
if "--print" in sys.argv:
print(table)
return 0
with open(DOC) as f:
text = f.read()
if START not in text or END not in text:
print(f"{DOC}: missing {START}/{END} markers", file=sys.stderr)
return 1
head, _, rest = text.partition(START)
_, _, tail = rest.partition(END)
with open(DOC, "w") as f:
f.write(head + table + tail)
print(f"{DOC}: key table rewritten from the registry ({len(KEYS)} keys)")
return 0
if __name__ == "__main__":
raise SystemExit(main())

14
tests/bluesky-pds/ops.py
View File

@ -9,14 +9,14 @@ sys.path.insert(0, os.path.dirname(__file__))
import _p4 # noqa: E402 import _p4 # noqa: E402
def pre_upgrade(domain, meta): def pre_upgrade(ctx):
_p4.create_account(domain) _p4.create_account(ctx.domain)
def pre_backup(domain, meta): def pre_backup(ctx):
_p4.create_account(domain) _p4.create_account(ctx.domain)
def pre_restore(domain, meta): def pre_restore(ctx):
_p4.delete_account(domain) _p4.delete_account(ctx.domain)
assert not _p4.account_exists(domain), "marker account delete did not take (pre_restore)" assert not _p4.account_exists(ctx.domain), "marker account delete did not take (pre_restore)"

143
tests/conftest.py
View File

@ -14,32 +14,7 @@ import pytest
sys.path.insert(0, os.path.join(os.path.dirname(__file__), "..", "runner")) sys.path.insert(0, os.path.join(os.path.dirname(__file__), "..", "runner"))
from harness import deps as deps_mod # noqa: E402 from harness import deps as deps_mod # noqa: E402
from harness import lifecycle, naming from harness import meta as meta_mod # noqa: E402
def _short(s: str, n: int = 8) -> str:
return "".join(c for c in s if c.isalnum())[:n] or "local"
def _recipe_meta(recipe: str) -> dict:
"""Optional per-recipe config so enrolling a recipe needs NO shared-harness change (D5).
A recipe may ship tests/<recipe>/recipe_meta.py with any of: HEALTH_PATH (str),
HEALTH_OK (tuple of status codes), DEPLOY_TIMEOUT (int), HTTP_TIMEOUT (int)."""
path = os.path.join(os.path.dirname(__file__), recipe, "recipe_meta.py")
meta = {
"HEALTH_PATH": "/",
"HEALTH_OK": (200, 301, 302),
"DEPLOY_TIMEOUT": 600,
"HTTP_TIMEOUT": 300,
}
if os.path.exists(path):
ns: dict = {}
with open(path) as fh:
exec(compile(fh.read(), path, "exec"), ns) # noqa: S102 (trusted, in-repo)
for k in meta:
if k in ns:
meta[k] = ns[k]
return meta
@pytest.fixture(scope="session") @pytest.fixture(scope="session")
@ -48,18 +23,10 @@ def recipe() -> str:
@pytest.fixture(scope="session") @pytest.fixture(scope="session")
def app_domain(recipe) -> str: def meta(recipe):
# Docker swarm config/secret names = <stackname>_<res>_<ver> must be <= 64 chars, and """The recipe's FULL validated customization (RecipeMeta, attribute access) via the single
# stackname is the sanitized domain. ".ci.commoninternet.net" alone is 22 chars, so the loader (rcust P1 — previously this fixture saw only the 4 base keys, spec §8 R3)."""
# subdomain label must stay short. Use <recipe[:4]>-<6hex(recipe|pr|ref)> — unique per run, return meta_mod.load(recipe)
# collision-safe across recipes (full recipe in the hash), readable context lives in the
# Drone build params + PR comment. (Deviation from plan §4.0 long name; see DECISIONS.md.)
return naming.app_domain(recipe, os.environ.get("PR", "0"), os.environ.get("REF"))
@pytest.fixture(scope="session")
def meta(recipe) -> dict:
return _recipe_meta(recipe)
@pytest.fixture(scope="session") @pytest.fixture(scope="session")
@ -73,32 +40,55 @@ def live_app() -> str:
return domain return domain
@pytest.fixture(scope="session") @pytest.fixture
def deps_apps() -> dict[str, str]: def op_state() -> dict:
"""Phase 2 Q2.3 dependency-resolver contract (refined operator-2026-05-28 SSO-dep plan §1): """The orchestrator's run-scoped op context (rcust P4): versions, artifact paths — written to
when a recipe declares `DEPS = [...]` in its `recipe_meta.py`, the orchestrator deploys each `$CCCI_OP_STATE_FILE` after each lifecycle op (e.g. `{"upgrade": {"before": {...},
dep AFTER the generic tiers (between RESTORE and CUSTOM) and persists their per-run identity "head_ref": ...}, "backup": {"snapshot_id": ...}}`). Overlay tests read op facts from here
+ SSO creds to `$CCCI_DEPS_FILE`. Tests access the dep's per-run domain via this fixture. instead of hand-parsing env/JSON. Skips with a clear reason outside an orchestrator run."""
For full SSO creds (realm/client/secret/admin) use the `deps_creds` fixture instead. import json
Returns `{dep_recipe: domain}` (str→str). Empty when no deps declared OR deps-not-ready.""" path = os.environ.get("CCCI_OP_STATE_FILE")
if not path:
pytest.skip(
"CCCI_OP_STATE_FILE not set — op_state is only available under the orchestrator"
)
if not os.path.exists(path):
pytest.skip(f"op-state file missing ({path}) — orchestrator has not performed an op yet")
try:
with open(path) as f:
return json.load(f)
except ValueError:
pytest.skip(f"op-state file unreadable/not JSON ({path})")
class _DepEntry(dict):
"""One provisioned dep (full creds dict) with attribute sugar: `entry.domain`, `entry.realm`,
`entry.client_secret`, ... — dict-style access works too (rcust P2d)."""
def __getattr__(self, name):
try:
return self[name]
except KeyError as e:
raise AttributeError(name) from e
@pytest.fixture(scope="session")
def deps() -> dict[str, _DepEntry]:
"""The recipe's provisioned deps (rcust P2d — consolidates the old `deps_apps`+`deps_creds`
pair). When a recipe declares `DEPS = [...]` in its `recipe_meta.py`, the orchestrator
provisions each dep BEFORE the single deploy and persists per-run identity + SSO creds to
`$CCCI_DEPS_FILE`. `deps["keycloak"]` carries domain/realm/client_id/client_secret/user/
password/email/admin_user/admin_password/discovery_url/token_url/... (`.domain` etc. work as
attributes). Empty when no deps declared OR deps-not-ready — pair with
`@pytest.mark.requires_deps` so the F2-11 skip-report keeps the green signal honest."""
state = deps_mod.deps_as_dict(deps_mod.load_run_state()) state = deps_mod.deps_as_dict(deps_mod.load_run_state())
return {r: e["domain"] for r, e in state.items() if e.get("domain")} return {r: _DepEntry(e) for r, e in state.items()}
@pytest.fixture(scope="session")
def deps_creds() -> dict[str, dict]:
"""Full SSO-creds dict for each declared dep (operator-2026-05-28 SSO-dep plan §1).
`deps_creds["keycloak"]` returns the entry written by setup_custom_tests with keys
domain/realm/client_id/client_secret/user/password/email/admin_user/admin_password/
discovery_url/token_url/.... Use this in `@pytest.mark.requires_deps` tests that need to
authenticate via OIDC."""
return deps_mod.deps_as_dict(deps_mod.load_run_state())
def pytest_collection_modifyitems(config, items): def pytest_collection_modifyitems(config, items):
"""SSO-dep plan §4: tests marked `@pytest.mark.requires_deps` are skipped with reason """SSO-dep plan §4: tests marked `@pytest.mark.requires_deps` are skipped with reason
`deps-not-ready: <captured-err>` when the orchestrator's setup_custom_tests step failed `deps-not-ready: <captured-err>` when the orchestrator's dep provisioning failed
(orchestrator sets CCCI_DEPS_READY=0 in env). Non-deps custom tests are unaffected. (orchestrator sets CCCI_DEPS_READY=0 in env). Non-deps custom tests are unaffected.
This is failure-isolation per plan §1 — generic tiers cannot break the SSO-marked tests' This is failure-isolation per plan §1 — generic tiers cannot break the SSO-marked tests'
@ -131,40 +121,5 @@ def pytest_configure(config):
"""Register the `requires_deps` marker so pytest doesn't warn about it.""" """Register the `requires_deps` marker so pytest doesn't warn about it."""
config.addinivalue_line( config.addinivalue_line(
"markers", "markers",
"requires_deps: test requires DEPS-declared services + setup_custom_tests success.", "requires_deps: test requires DEPS-declared services + dep provisioning success.",
) )
def _wait_healthy(domain, meta):
lifecycle.wait_healthy(
domain,
ok_codes=tuple(meta["HEALTH_OK"]),
path=meta["HEALTH_PATH"],
deploy_timeout=meta["DEPLOY_TIMEOUT"],
http_timeout=meta["HTTP_TIMEOUT"],
)
@pytest.fixture
def deployed(recipe, app_domain, meta, request):
"""Function-scoped: deploy the current/$REF version healthy, guaranteed teardown after.
Used by stages that start from current (install/backup)."""
version = os.environ.get("VERSION") or None
lifecycle.janitor()
request.addfinalizer(lambda: lifecycle.teardown_app(app_domain))
lifecycle.deploy_app(recipe, app_domain, version=version)
_wait_healthy(app_domain, meta)
return app_domain
@pytest.fixture(scope="session")
def deployed_app(recipe, app_domain, meta):
"""Install stage: deploy the recipe and wait until healthy; tear down at session end."""
version = os.environ.get("VERSION") or None
lifecycle.janitor() # sweep orphans from crashed runs first
try:
lifecycle.deploy_app(recipe, app_domain, version=version, secrets=True)
_wait_healthy(app_domain, meta)
yield app_domain
finally:
lifecycle.teardown_app(app_domain)

12
tests/cryptpad/ops.py
View File

@ -15,13 +15,13 @@ def _write(domain, val):
lifecycle.exec_in_app(domain, ["sh", "-c", f"echo {val} > {MARKER}"]) lifecycle.exec_in_app(domain, ["sh", "-c", f"echo {val} > {MARKER}"])
def pre_upgrade(domain, meta): def pre_upgrade(ctx):
_write(domain, "upgrade-survives") _write(ctx.domain, "upgrade-survives")
def pre_backup(domain, meta): def pre_backup(ctx):
_write(domain, "original") _write(ctx.domain, "original")
def pre_restore(domain, meta): def pre_restore(ctx):
_write(domain, "mutated") # diverge so a successful restore is observable _write(ctx.domain, "mutated") # diverge so a successful restore is observable

4
tests/cryptpad/recipe_meta.py
View File

@ -7,9 +7,9 @@ DEPLOY_TIMEOUT = 600
HTTP_TIMEOUT = 600 HTTP_TIMEOUT = 600
def EXTRA_ENV(domain): def EXTRA_ENV(ctx):
"""cryptpad needs a SANDBOX_DOMAIN distinct from the main DOMAIN (it serves user content from a """cryptpad needs a SANDBOX_DOMAIN distinct from the main DOMAIN (it serves user content from a
separate origin; the web router routes both). Derive a sibling subdomain under the same wildcard separate origin; the web router routes both). Derive a sibling subdomain under the same wildcard
(covered by the wildcard cert, so no cert work).""" (covered by the wildcard cert, so no cert work)."""
label, _, rest = domain.partition(".") label, _, rest = ctx.domain.partition(".")
return {"SANDBOX_DOMAIN": f"{label}-sb.{rest}"} return {"SANDBOX_DOMAIN": f"{label}-sb.{rest}"}

4
tests/custom-html-bkp-bad/ops.py
View File

@ -12,8 +12,8 @@ from harness import lifecycle
MARKER_PATH = "/usr/share/nginx/html/ci-marker.txt" MARKER_PATH = "/usr/share/nginx/html/ci-marker.txt"
def pre_restore(domain: str, meta: dict) -> None: def pre_restore(ctx) -> None:
"""Write 'mutated' to the marker before restore runs. If restore brings back the """Write 'mutated' to the marker before restore runs. If restore brings back the
snapshot (which has no marker — never seeded by pre_backup), the marker ends up snapshot (which has no marker — never seeded by pre_backup), the marker ends up
MISSING or 'mutated' after restore → test_restore_returns_state FAILS → restore=RED.""" MISSING or 'mutated' after restore → test_restore_returns_state FAILS → restore=RED."""
lifecycle.exec_in_app(domain, ["sh", "-c", f"echo mutated > {MARKER_PATH}"]) lifecycle.exec_in_app(ctx.domain, ["sh", "-c", f"echo mutated > {MARKER_PATH}"])

4
tests/custom-html-rst-bad/ops.py
View File

@ -11,5 +11,5 @@ from harness import lifecycle
MARKER_PATH = "/usr/share/nginx/html/ci-marker.txt" MARKER_PATH = "/usr/share/nginx/html/ci-marker.txt"
def pre_restore(domain: str, meta: dict) -> None: def pre_restore(ctx) -> None:
lifecycle.exec_in_app(domain, ["sh", "-c", f"echo mutated > {MARKER_PATH}"]) lifecycle.exec_in_app(ctx.domain, ["sh", "-c", f"echo mutated > {MARKER_PATH}"])

14
tests/custom-html/ops.py
View File

@ -1,4 +1,4 @@
"""custom-html — pre-op seed hooks (Phase 1e HC3). The orchestrator runs `pre_<op>(domain, meta)` """custom-html — pre-op seed hooks (Phase 1e HC3). The orchestrator runs `pre_<op>(ctx)`
BEFORE it performs the op; the matching test_<op>.py asserts the post-op state (assertion-only). BEFORE it performs the op; the matching test_<op>.py asserts the post-op state (assertion-only).
nginx serves the volume at /usr/share/nginx/html, so the marker file survives an upgrade / a nginx serves the volume at /usr/share/nginx/html, so the marker file survives an upgrade / a
@ -17,16 +17,16 @@ def _write(domain: str, val: str) -> None:
lifecycle.exec_in_app(domain, ["sh", "-c", f"echo {val} > {MARKER_PATH}"]) lifecycle.exec_in_app(domain, ["sh", "-c", f"echo {val} > {MARKER_PATH}"])
def pre_upgrade(domain, meta): def pre_upgrade(ctx):
# seed a marker before the upgrade so the overlay can prove the data survives it # seed a marker before the upgrade so the overlay can prove the data survives it
_write(domain, "upgrade-survives") _write(ctx.domain, "upgrade-survives")
def pre_backup(domain, meta): def pre_backup(ctx):
# establish a known original state before the backup op captures it # establish a known original state before the backup op captures it
_write(domain, "original") _write(ctx.domain, "original")
def pre_restore(domain, meta): def pre_restore(ctx):
# diverge from the backed-up state so a successful restore (back to "original") is observable # diverge from the backed-up state so a successful restore (back to "original") is observable
_write(domain, "mutated") _write(ctx.domain, "mutated")

28
tests/discourse/install_steps.sh
View File

@ -1,28 +0,0 @@
#!/usr/bin/env bash
# discourse — INSTALL-TIME hook (Phase 2 Q4.6). Runs during the install tier AFTER `abra app new` +
# EXTRA_ENV + `abra app secret generate` and BEFORE the single `abra app deploy`
# (lifecycle.py::_run_install_steps), with CCCI_RECIPE / CCCI_APP_DOMAIN in env.
#
# Purpose: provide the cc-ci re-pin+grace overlay (compose.ccci.yml) to the recipe checkout so the
# UPGRADE-tier BASE deploy (published 0.7.0+3.3.1, whose compose pins the Docker-Hub-removed
# `bitnami/discourse:3.3.1` and ships a too-tight 5m start_period) is deployable and can survive the
# 15-25min Rails cold boot — so upgrade-to-latest can run. See compose.ccci.yml's header for the full
# rationale. The overlay is referenced by recipe_meta COMPOSE_FILE; it is a cc-ci file (not part of the
# recipe), so copying it here makes it resolvable. It persists across the later `git checkout <head>`
# (untracked) so the head deploy also merges it (idempotent — the PR head already re-pins + ships 20m).
# CHAOS_BASE_DEPLOY=True is set so abra's pinned-deploy clean-tree check doesn't FATA on the overlay.
set -euo pipefail
: "${CCCI_RECIPE:?missing CCCI_RECIPE}"
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
# Resolve the recipe tree the way abra does: $ABRA_DIR (the per-run tree inside a CI run) else
# the canonical ~/.abra — the overlay must land in the tree this run actually deploys from.
RECIPE_DIR="${ABRA_DIR:-${HOME}/.abra}/recipes/${CCCI_RECIPE}"
if [ ! -d "$RECIPE_DIR" ]; then
echo " discourse install_steps: recipe dir $RECIPE_DIR missing — cannot provide compose.ccci.yml" >&2
exit 1
fi
cp "$SCRIPT_DIR/compose.ccci.yml" "$RECIPE_DIR/compose.ccci.yml"
echo " discourse install_steps: provided compose.ccci.yml (bitnamilegacy re-pin + 20m start_period grace) to recipe checkout (${CCCI_RECIPE})"

14
tests/discourse/ops.py
View File

@ -30,18 +30,18 @@ def _seed(domain, value):
assert got == value, f"seed did not commit (read back {got!r}, expected {value!r})" assert got == value, f"seed did not commit (read back {got!r}, expected {value!r})"
def pre_upgrade(domain, meta): def pre_upgrade(ctx):
_seed(domain, "upgrade-survives") _seed(ctx.domain, "upgrade-survives")
def pre_backup(domain, meta): def pre_backup(ctx):
_seed(domain, "original") _seed(ctx.domain, "original")
def pre_restore(domain, meta): def pre_restore(ctx):
# diverge from the backup so a successful restore is observable # diverge from the backup so a successful restore is observable
_psql(domain, "DROP TABLE IF EXISTS ci_marker;") _psql(ctx.domain, "DROP TABLE IF EXISTS ci_marker;")
assert _psql(domain, "SELECT to_regclass('public.ci_marker');") in ( assert _psql(ctx.domain, "SELECT to_regclass('public.ci_marker');") in (
"", "",
"NULL", "NULL",
), "drop did not take" ), "drop did not take"

10
tests/discourse/recipe_meta.py
View File

@ -29,11 +29,11 @@ HTTP_TIMEOUT = 1200
# (1) it pins the Docker-Hub-removed `bitnami/discourse:3.3.1` (404) → overlay re-pins app+sidekiq to # (1) it pins the Docker-Hub-removed `bitnami/discourse:3.3.1` (404) → overlay re-pins app+sidekiq to
# `bitnamilegacy/discourse:3.3.1` (namespace-only, identical image), the same re-pin the PR makes; # `bitnamilegacy/discourse:3.3.1` (namespace-only, identical image), the same re-pin the PR makes;
# (2) its 5m start_period is too tight for the 15-25min Rails boot → overlay widens it to 20m (grace). # (2) its 5m start_period is too tight for the 15-25min Rails boot → overlay widens it to 20m (grace).
# install_steps.sh provides the overlay; CHAOS_BASE_DEPLOY skips the clean-tree gate on the untracked # The harness auto-provides the overlay to the checkout and auto-chaoses the base deploy
# overlay; it persists across the head checkout (idempotent — the PR head already re-pins + ships 20m). # (first-class compose.ccci.yml, rcust P2a); it persists across the head checkout (idempotent — the
# PR head already re-pins + ships 20m).
# Upgrade crossover: 0.7.0 (re-pinned base) → PR head; full assertions run on the HEAD. The 0.7.0 # Upgrade crossover: 0.7.0 (re-pinned base) → PR head; full assertions run on the HEAD. The 0.7.0
# *custom* tests are not separately run (custom tier runs once, on the head — policy §1 allows skip+record). # *custom* tests are not separately run (custom tier runs once, on the head — policy §1 allows skip+record).
CHAOS_BASE_DEPLOY = True
UPGRADE_BASE_VERSION = "0.7.0+3.3.1" UPGRADE_BASE_VERSION = "0.7.0+3.3.1"
EXTRA_ENV = { EXTRA_ENV = {
"TIMEOUT": "3600", # abra's internal convergence wait; matches DEPLOY_TIMEOUT (slow Rails boot headroom) "TIMEOUT": "3600", # abra's internal convergence wait; matches DEPLOY_TIMEOUT (slow Rails boot headroom)
@ -41,7 +41,7 @@ EXTRA_ENV = {
} }
def BACKUP_VERIFY(domain): def BACKUP_VERIFY(ctx):
"""Post-backup integrity check (Q4.6, same race ghost F2-14b hit). The recipe's backupbot db """Post-backup integrity check (Q4.6, same race ghost F2-14b hit). The recipe's backupbot db
pre-hook (`/pg_backup.sh backup`) dumps the discourse postgres DB to `/var/lib/postgresql/data/ pre-hook (`/pg_backup.sh backup`) dumps the discourse postgres DB to `/var/lib/postgresql/data/
backup.sql` (gzip), then restic captures that path. On the loaded single CI node the db container backup.sql` (gzip), then restic captures that path. On the loaded single CI node the db container
@ -60,7 +60,7 @@ def BACKUP_VERIFY(domain):
try: try:
out = lifecycle.exec_in_app( out = lifecycle.exec_in_app(
domain, ctx.domain,
[ [
"sh", "sh",
"-c", "-c",

28
tests/ghost/install_steps.sh
View File

@ -1,28 +0,0 @@
#!/usr/bin/env bash
# ghost — INSTALL-TIME hook (Phase 2 F2-14b). Runs during the install tier AFTER `abra app new` +
# EXTRA_ENV + `abra app secret generate` and BEFORE the single `abra app deploy`
# (lifecycle.py::_run_install_steps), with CCCI_RECIPE / CCCI_APP_DOMAIN in env.
#
# Purpose: provide the cc-ci start_period-grace overlay (compose.ccci.yml) to the recipe checkout so
# the UPGRADE-tier BASE deploy (a previous published version whose app healthcheck still ships the
# too-tight 1m start_period) can survive ghost's ~6-9min fresh-DB migration and converge. See
# compose.ccci.yml's header for the full rationale. The overlay is referenced by recipe_meta
# COMPOSE_FILE; copying it here (it is a cc-ci file, not part of the recipe) makes it resolvable.
# It persists across the later `git checkout <head>` (untracked) so the head deploy also merges it
# (idempotent — the PR head already ships 15m). CHAOS_BASE_DEPLOY=True is set so abra's pinned-deploy
# clean-tree check doesn't FATA on the untracked overlay.
set -euo pipefail
: "${CCCI_RECIPE:?missing CCCI_RECIPE}"
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
# Resolve the recipe tree the way abra does: $ABRA_DIR (the per-run tree inside a CI run) else
# the canonical ~/.abra — the overlay must land in the tree this run actually deploys from.
RECIPE_DIR="${ABRA_DIR:-${HOME}/.abra}/recipes/${CCCI_RECIPE}"
if [ ! -d "$RECIPE_DIR" ]; then
echo " ghost install_steps: recipe dir $RECIPE_DIR missing — cannot provide compose.ccci.yml" >&2
exit 1
fi
cp "$SCRIPT_DIR/compose.ccci.yml" "$RECIPE_DIR/compose.ccci.yml"
echo " ghost install_steps: provided compose.ccci.yml (app start_period grace) to recipe checkout (${CCCI_RECIPE})"

14
tests/ghost/ops.py
View File

@ -36,19 +36,19 @@ def _seed(domain, value):
assert got == value, f"seed did not commit (read back {got!r}, expected {value!r})" assert got == value, f"seed did not commit (read back {got!r}, expected {value!r})"
def pre_upgrade(domain, meta): def pre_upgrade(ctx):
_seed(domain, "upgrade-survives") _seed(ctx.domain, "upgrade-survives")
def pre_backup(domain, meta): def pre_backup(ctx):
_seed(domain, "original") _seed(ctx.domain, "original")
def pre_restore(domain, meta): def pre_restore(ctx):
# diverge from the backup so a successful restore is observable: drop the marker table. # diverge from the backup so a successful restore is observable: drop the marker table.
_mysql(domain, "DROP TABLE IF EXISTS ci_marker;") _mysql(ctx.domain, "DROP TABLE IF EXISTS ci_marker;")
got = _mysql( got = _mysql(
domain, ctx.domain,
"SELECT COUNT(*) FROM information_schema.tables " "SELECT COUNT(*) FROM information_schema.tables "
"WHERE table_schema='ghost' AND table_name='ci_marker';", "WHERE table_schema='ghost' AND table_name='ci_marker';",
) )

11
tests/ghost/recipe_meta.py
View File

@ -31,23 +31,22 @@ HTTP_TIMEOUT = 900
# (plan-ccci-compose-overlay-policy.md §1), so the harness base-deploys the previous PUBLISHED version # (plan-ccci-compose-overlay-policy.md §1), so the harness base-deploys the previous PUBLISHED version
# (1.1.1+6-alpine) — which predates the PR and still ships the too-tight 1m start_period → it would # (1.1.1+6-alpine) — which predates the PR and still ships the too-tight 1m start_period → it would
# deadlock on the same migration kill. compose.ccci.yml re-applies the 15m grace to the BASE so the # deadlock on the same migration kill. compose.ccci.yml re-applies the 15m grace to the BASE so the
# from-version is deployable; install_steps.sh provides it to the checkout; CHAOS_BASE_DEPLOY skips the # from-version is deployable; the harness auto-provides it to the checkout and auto-chaoses the base
# clean-tree gate on that untracked overlay. It persists across the head checkout (idempotent — the PR # deploy (first-class compose.ccci.yml, rcust P2a). It persists across the head checkout (idempotent —
# head already ships 15m). This is the policy-blessed "minimal overlay on the from-version so # the PR head already ships 15m). This is the policy-blessed "minimal overlay on the from-version so
# upgrade-to-latest can run" — grace-only, masks no defect, weakens no test. # upgrade-to-latest can run" — grace-only, masks no defect, weakens no test.
# TIMEOUT/DEPLOY_TIMEOUT 2400s: the BASE cold boot's wall-time is mysql fresh-dir init (~6min, during # TIMEOUT/DEPLOY_TIMEOUT 2400s: the BASE cold boot's wall-time is mysql fresh-dir init (~6min, during
# which the app crash-loops harmlessly on `ECONNREFUSED 3306` until mysql accepts connections — no # which the app crash-loops harmlessly on `ECONNREFUSED 3306` until mysql accepts connections — no
# migration progress lost, it hasn't started) PLUS the ~9-15min schema migration (round-trip-bound, # migration progress lost, it hasn't started) PLUS the ~9-15min schema migration (round-trip-bound,
# slower under host load). 1200s was too tight (full4 killed at the near-final `email_recipients` # slower under host load). 1200s was too tight (full4 killed at the near-final `email_recipients`
# tables while still 0/1); 2400s gives headroom while still bounding a genuine hang (matches discourse). # tables while still 0/1); 2400s gives headroom while still bounding a genuine hang (matches discourse).
CHAOS_BASE_DEPLOY = True
EXTRA_ENV = { EXTRA_ENV = {
"TIMEOUT": "2400", "TIMEOUT": "2400",
"COMPOSE_FILE": "compose.yml:compose.ccci.yml", "COMPOSE_FILE": "compose.yml:compose.ccci.yml",
} }
def BACKUP_VERIFY(domain): def BACKUP_VERIFY(ctx):
"""Post-backup integrity check (F2-14b). The recipe's backupbot db pre-hook dumps the ghost MySQL """Post-backup integrity check (F2-14b). The recipe's backupbot db pre-hook dumps the ghost MySQL
DB to `/var/lib/mysql/backup.sql.gz` (then restic captures that path). On the loaded single CI node DB to `/var/lib/mysql/backup.sql.gz` (then restic captures that path). On the loaded single CI node
the db container intermittently CYCLES mid-dump (observed: full5/6/7 RED, full8 green — pure race; the db container intermittently CYCLES mid-dump (observed: full5/6/7 RED, full8 green — pure race;
@ -62,7 +61,7 @@ def BACKUP_VERIFY(domain):
try: try:
out = lifecycle.exec_in_app( out = lifecycle.exec_in_app(
domain, ctx.domain,
[ [
"sh", "sh",
"-c", "-c",

14
tests/immich/ops.py
View File

@ -25,17 +25,17 @@ def _seed(domain, value):
assert _psql(domain, "SELECT v FROM ci_marker;") == value assert _psql(domain, "SELECT v FROM ci_marker;") == value
def pre_upgrade(domain, meta): def pre_upgrade(ctx):
_seed(domain, "upgrade-survives") _seed(ctx.domain, "upgrade-survives")
def pre_backup(domain, meta): def pre_backup(ctx):
_seed(domain, "original") _seed(ctx.domain, "original")
def pre_restore(domain, meta): def pre_restore(ctx):
_psql(domain, "DROP TABLE ci_marker;") _psql(ctx.domain, "DROP TABLE ci_marker;")
assert _psql(domain, "SELECT to_regclass('public.ci_marker');") in ( assert _psql(ctx.domain, "SELECT to_regclass('public.ci_marker');") in (
"", "",
"NULL", "NULL",
), "drop did not take" ), "drop did not take"

18
tests/keycloak/ops.py
View File

@ -14,20 +14,20 @@ def _token(domain):
return kc_admin.admin_token(domain, kc_admin.admin_password(domain)) return kc_admin.admin_token(domain, kc_admin.admin_password(domain))
def pre_upgrade(domain, meta): def pre_upgrade(ctx):
# create the marker realm (DB data) before the upgrade so the overlay can prove it survives # create the marker realm (DB data) before the upgrade so the overlay can prove it survives
assert kc_admin.create_marker_realm(domain, _token(domain)) in (201, 409) assert kc_admin.create_marker_realm(ctx.domain, _token(ctx.domain)) in (201, 409)
def pre_backup(domain, meta): def pre_backup(ctx):
# establish the marker realm before the backup op captures mariadb # establish the marker realm before the backup op captures mariadb
assert kc_admin.create_marker_realm(domain, _token(domain)) in (201, 409) assert kc_admin.create_marker_realm(ctx.domain, _token(ctx.domain)) in (201, 409)
def pre_restore(domain, meta): def pre_restore(ctx):
# backup-bot-two cycles the keycloak container during backup → wait for serving, re-auth, then # backup-bot-two cycles the keycloak container during backup → wait for serving, re-auth, then
# delete the realm (diverge from the backup) so a successful restore is observable # delete the realm (diverge from the backup) so a successful restore is observable
generic.assert_serving(domain, meta) generic.assert_serving(ctx.domain, ctx.meta)
tok = _token(domain) tok = _token(ctx.domain)
assert kc_admin.delete_marker_realm(domain, tok) in (204, 200) assert kc_admin.delete_marker_realm(ctx.domain, tok) in (204, 200)
assert not kc_admin.marker_realm_exists(domain, tok), "delete did not take" assert not kc_admin.marker_realm_exists(ctx.domain, tok), "delete did not take"

8
tests/lasuite-docs/functional/test_create_doc.py
View File

@ -5,7 +5,7 @@ persistence". This is the canonical create-an-object + read-it-back for lasuite-
Flow (uses an OIDC token from the dep keycloak): Flow (uses an OIDC token from the dep keycloak):
1. Obtain a JWT via OIDC password grant against the dep keycloak (the test user is provisioned 1. Obtain a JWT via OIDC password grant against the dep keycloak (the test user is provisioned
by the orchestrator's setup_custom_tests step). by the orchestrator's dep-provisioning step).
2. POST `/api/v1.0/documents/` with `Authorization: Bearer <jwt>` to create a new doc with a 2. POST `/api/v1.0/documents/` with `Authorization: Bearer <jwt>` to create a new doc with a
unique title; capture the returned `id`. unique title; capture the returned `id`.
3. GET `/api/v1.0/documents/<id>/` with the same Bearer token; assert the returned title and 3. GET `/api/v1.0/documents/<id>/` with the same Bearer token; assert the returned title and
@ -15,7 +15,7 @@ Non-vacuous: a misconfigured OIDC, broken backend, or missing endpoint fails at
broken. The marker-in-the-title + id round-trip proves the doc actually persisted in lasuite- broken. The marker-in-the-title + id round-trip proves the doc actually persisted in lasuite-
docs's database after going through the recipe's nginx → backend → postgres path. docs's database after going through the recipe's nginx → backend → postgres path.
Marked @pytest.mark.requires_deps — skips with `deps-not-ready` if setup_custom_tests failed. Marked @pytest.mark.requires_deps — skips with `deps-not-ready` if dep provisioning failed.
""" """
from __future__ import annotations from __future__ import annotations
@ -32,9 +32,9 @@ from harness import sso
@pytest.mark.requires_deps @pytest.mark.requires_deps
def test_create_doc_and_read_back(live_app, deps_creds): def test_create_doc_and_read_back(live_app, deps):
"""Create a doc via the authenticated API; fetch it back; assert round-trip.""" """Create a doc via the authenticated API; fetch it back; assert round-trip."""
kc = deps_creds["keycloak"] kc = deps["keycloak"]
# Obtain a JWT via OIDC password grant # Obtain a JWT via OIDC password grant
access_token = sso.oidc_password_grant( access_token = sso.oidc_password_grant(

8
tests/lasuite-docs/functional/test_oidc_login.py
View File

@ -5,13 +5,13 @@ SOURCE: references/recipe-maintainer/recipe-info/lasuite-docs/tests/oidc_login.p
End-to-end flow: End-to-end flow:
1. GET `/api/v1.0/users/me/` without auth → asserts the response REDIRECTS to the dep 1. GET `/api/v1.0/users/me/` without auth → asserts the response REDIRECTS to the dep
keycloak's realm auth endpoint (the recipe is correctly configured to challenge keycloak's realm auth endpoint (the recipe is correctly configured to challenge
unauthenticated callers — wired via setup_custom_tests.sh). unauthenticated callers — wired via install_steps.sh).
2. Obtain an OIDC token from the dep keycloak via password grant 2. Obtain an OIDC token from the dep keycloak via password grant
(the test user provisioned by the orchestrator's realm setup). (the test user provisioned by the orchestrator's realm setup).
3. Call `/api/v1.0/users/me/` with `Authorization: Bearer <jwt>` → asserts 200 and the 3. Call `/api/v1.0/users/me/` with `Authorization: Bearer <jwt>` → asserts 200 and the
returned user's email matches the provisioned test user. returned user's email matches the provisioned test user.
Marked @pytest.mark.requires_deps — skips with `deps-not-ready` if setup_custom_tests failed. Marked @pytest.mark.requires_deps — skips with `deps-not-ready` if dep provisioning failed.
""" """
from __future__ import annotations from __future__ import annotations
@ -51,9 +51,9 @@ def _get_no_redirect(url: str) -> tuple[int, str]:
@pytest.mark.requires_deps @pytest.mark.requires_deps
def test_oidc_login_via_keycloak(live_app, deps_creds): def test_oidc_login_via_keycloak(live_app, deps):
"""Anonymous → redirect to keycloak; password-grant token → 200 from /api/v1.0/users/me/.""" """Anonymous → redirect to keycloak; password-grant token → 200 from /api/v1.0/users/me/."""
kc = deps_creds["keycloak"] kc = deps["keycloak"]
# Step 1: unauthenticated GET → 302 to keycloak realm's auth endpoint # Step 1: unauthenticated GET → 302 to keycloak realm's auth endpoint
status, redirect = _get_no_redirect(f"https://{live_app}/api/v1.0/users/me/") status, redirect = _get_no_redirect(f"https://{live_app}/api/v1.0/users/me/")

16
tests/lasuite-docs/functional/test_oidc_with_keycloak.py
View File

@ -3,10 +3,10 @@
Refactored to the refined SSO-dep model: Refactored to the refined SSO-dep model:
- The orchestrator deploys a per-run keycloak dep AFTER generic tiers and provisions a fresh - The orchestrator deploys a per-run keycloak dep AFTER generic tiers and provisions a fresh
realm/client/user via `harness.sso.setup_keycloak_realm`. The creds are written to realm/client/user via `harness.sso.setup_keycloak_realm`. The creds are written to
`$CCCI_DEPS_FILE` (read here via the `deps_creds` fixture). `$CCCI_DEPS_FILE` (read here via the `deps` fixture).
- This test no longer calls `setup_keycloak_realm` itself — that's the orchestrator's job in - This test no longer calls `setup_keycloak_realm` itself — that's the orchestrator's job in
the setup_custom_tests step. We just consume the credentials and exercise the OIDC flow. the dep-provisioning step. We just consume the credentials and exercise the OIDC flow.
- Marked `@pytest.mark.requires_deps` so if setup_custom_tests failed, this test SKIPs with a - Marked `@pytest.mark.requires_deps` so if dep provisioning failed, this test SKIPs with a
clear `deps-not-ready` reason rather than red-flagging a non-recipe failure. clear `deps-not-ready` reason rather than red-flagging a non-recipe failure.
""" """
@ -31,13 +31,13 @@ def _b64url_decode(seg: str) -> bytes:
@pytest.mark.requires_deps @pytest.mark.requires_deps
def test_oidc_password_grant_against_dep_keycloak(live_app, deps_creds): def test_oidc_password_grant_against_dep_keycloak(live_app, deps):
"""The dep keycloak issues a JWT for the pre-provisioned test user via OIDC password grant.""" """The dep keycloak issues a JWT for the pre-provisioned test user via OIDC password grant."""
assert "keycloak" in deps_creds, ( assert "keycloak" in deps, (
f"keycloak creds not in deps_creds; got {list(deps_creds.keys())}. " f"keycloak creds not in deps; got {list(deps.keys())}. "
"setup_custom_tests should have populated this." "dep provisioning should have populated this."
) )
kc = deps_creds["keycloak"] kc = deps["keycloak"]
# Sanity-check the creds shape — orchestrator-written # Sanity-check the creds shape — orchestrator-written
assert kc["domain"] assert kc["domain"]

74
tests/lasuite-docs/install_steps.sh Executable file
View File

@ -0,0 +1,74 @@
#!/usr/bin/env bash
# lasuite-docs — INSTALL-TIME OIDC wiring hook (rcust P2b; migrated from the deleted
# setup_custom_tests.sh post-deploy path — sibling of lasuite-drive/-meet's hooks).
#
# Runs during the install tier AFTER `abra app new` + EXTRA_ENV + `abra app secret generate`, and
# BEFORE the single `abra app deploy` (lifecycle.py::_run_install_steps). Writing OIDC env + the
# real client secret HERE means the recipe deploys ONCE with OIDC already wired — no post-deploy
# reconverge. The orchestrator provisions the per-run realm/client on the (live-warm) keycloak
# BEFORE this hook and writes $CCCI_DEPS_FILE (the recipe→creds dict). docs' OIDC settings are
# config-only (validated by `manage.py check`, not fetched at boot), so the stack boots healthy
# with the env set. Env names per lasuite-docs's .env.sample (same values the old post-deploy
# hook wrote — byte-identical wiring, only the timing moved).
#
# Env supplied by the harness:
# CCCI_APP_DOMAIN — the per-run lasuite-docs app domain
# CCCI_APP_ENV — path to the app's .env (the one `abra app deploy` reads)
# CCCI_DEPS_FILE — JSON {keycloak: {domain, realm, client_id, client_secret, ...}} (may be empty)
set -euo pipefail
: "${CCCI_APP_DOMAIN:?missing}"
ENV_PATH="${CCCI_APP_ENV:?missing}"
# No deps file / no keycloak entry → install-time provisioning failed or was skipped. NO-OP so the
# recipe still boots; the @requires_deps OIDC custom test then SKIPs and F2-11 flips the run RED.
if [ -z "${CCCI_DEPS_FILE:-}" ] || [ ! -s "${CCCI_DEPS_FILE}" ]; then
echo " install_steps: no deps file — skipping OIDC wiring (recipe boots without OIDC)"
exit 0
fi
KC_DOMAIN=$(jq -r '.keycloak.domain // empty' "$CCCI_DEPS_FILE")
KC_REALM=$(jq -r '.keycloak.realm // empty' "$CCCI_DEPS_FILE")
KC_CLIENT=$(jq -r '.keycloak.client_id // empty' "$CCCI_DEPS_FILE")
KC_SECRET=$(jq -r '.keycloak.client_secret // empty' "$CCCI_DEPS_FILE")
if [ -z "$KC_DOMAIN" ] || [ -z "$KC_SECRET" ]; then
echo " install_steps: deps file has no keycloak domain/secret — skipping OIDC wiring"
exit 0
fi
echo " lasuite-docs install_steps: wiring OIDC at install against keycloak ${KC_DOMAIN}"
# 1) Insert the OIDC client secret at a bumped version (abra already generated oidc_rpcs:v1; swarm
# forbids overwriting a secret at the same version). The app is not deployed yet — a swarm secret
# can be created independently — so the single deploy below picks up v2.
CUR_VER=$(grep -E '^\s*SECRET_OIDC_RPCS_VERSION=' "$ENV_PATH" | tail -1 | cut -d= -f2 | tr -d '"\r' || echo "v1")
NEW_NUM=$((${CUR_VER#v} + 1))
NEW_VER="v${NEW_NUM}"
INSERT_LOG=$(abra app secret insert "$CCCI_APP_DOMAIN" oidc_rpcs "$NEW_VER" "$KC_SECRET" --no-input -C -o 2>&1) ||
INSERT_LOG=$(script -qec "abra app secret insert $CCCI_APP_DOMAIN oidc_rpcs $NEW_VER $KC_SECRET --no-input -C -o" /dev/null 2>&1) ||
{
echo " install_steps: abra app secret insert oidc_rpcs@$NEW_VER failed: $INSERT_LOG"
exit 1
}
sed -i "s|^\s*SECRET_OIDC_RPCS_VERSION=.*|SECRET_OIDC_RPCS_VERSION=$NEW_VER|" "$ENV_PATH"
echo " install_steps: oidc_rpcs secret inserted at $NEW_VER (was $CUR_VER)"
# 2) Write OIDC env vars to the app's .env (names per lasuite-docs's .env.sample). Ensure a
# trailing newline first so appends never concatenate onto the last line.
write_env() {
local key="$1" val="$2"
sed -i "/^\s*#\?\s*${key}=/d" "$ENV_PATH"
[ -z "$(tail -c1 "$ENV_PATH" 2>/dev/null)" ] || printf '\n' >>"$ENV_PATH"
printf '%s=%s\n' "$key" "$val" >>"$ENV_PATH"
}
write_env OIDC_REALM "$KC_REALM"
write_env OIDC_OP_DISCOVERY_ENDPOINT "https://${KC_DOMAIN}/realms/${KC_REALM}/.well-known/openid-configuration"
write_env OIDC_OP_AUTHORIZATION_ENDPOINT "https://${KC_DOMAIN}/realms/${KC_REALM}/protocol/openid-connect/auth"
write_env OIDC_OP_TOKEN_ENDPOINT "https://${KC_DOMAIN}/realms/${KC_REALM}/protocol/openid-connect/token"
write_env OIDC_OP_USER_ENDPOINT "https://${KC_DOMAIN}/realms/${KC_REALM}/protocol/openid-connect/userinfo"
write_env OIDC_OP_LOGOUT_ENDPOINT "https://${KC_DOMAIN}/realms/${KC_REALM}/protocol/openid-connect/logout"
write_env OIDC_OP_JWKS_ENDPOINT "https://${KC_DOMAIN}/realms/${KC_REALM}/protocol/openid-connect/certs"
write_env OIDC_RP_CLIENT_ID "$KC_CLIENT"
write_env OIDC_RP_SIGN_ALGO "RS256"
write_env OIDC_RP_SCOPES "openid email profile"
echo " lasuite-docs install_steps: OIDC env wired into .env (deploy will pick it up, no reconverge)"

14
tests/lasuite-docs/ops.py
View File

@ -24,18 +24,18 @@ def _seed(domain, value):
assert _psql(domain, "SELECT v FROM ci_marker;") == value assert _psql(domain, "SELECT v FROM ci_marker;") == value
def pre_upgrade(domain, meta): def pre_upgrade(ctx):
_seed(domain, "upgrade-survives") _seed(ctx.domain, "upgrade-survives")
def pre_backup(domain, meta): def pre_backup(ctx):
_seed(domain, "original") _seed(ctx.domain, "original")
def pre_restore(domain, meta): def pre_restore(ctx):
# drop the marker table (diverge from the backup) so a successful restore is observable # drop the marker table (diverge from the backup) so a successful restore is observable
_psql(domain, "DROP TABLE ci_marker;") _psql(ctx.domain, "DROP TABLE ci_marker;")
assert _psql(domain, "SELECT to_regclass('public.ci_marker');") in ( assert _psql(ctx.domain, "SELECT to_regclass('public.ci_marker');") in (
"", "",
"NULL", "NULL",
), "drop did not take" ), "drop did not take"

2
tests/lasuite-docs/recipe_meta.py
View File

@ -15,7 +15,7 @@ HTTP_TIMEOUT = 600
DEPS = ["keycloak"] DEPS = ["keycloak"]
def EXTRA_ENV(domain): def EXTRA_ENV(ctx):
# abra's internal per-deploy convergence timeout (the recipe's TIMEOUT env, default 300s) is too # abra's internal per-deploy convergence timeout (the recipe's TIMEOUT env, default 300s) is too
# short for this 9-service stack on a COLD image cache (~9 large images: impress frontend/backend, # short for this 9-service stack on a COLD image cache (~9 large images: impress frontend/backend,
# minio, postgres18, redis, docspec, y-provider). Cold pulls exceed 300s -> "deploy timed out 🟠". # minio, postgres18, redis, docspec, y-provider). Cold pulls exceed 300s -> "deploy timed out 🟠".

91
tests/lasuite-docs/setup_custom_tests.sh
View File

@ -1,91 +0,0 @@
#!/usr/bin/env bash
# lasuite-docs — post-deps setup hook (operator-2026-05-28 SSO-dep plan §3.2).
#
# Runs AFTER the generic tiers (install/upgrade/backup/restore) and AFTER each declared dep is
# deployed + provisioned with realm/client via the harness. The orchestrator has written
# $CCCI_DEPS_FILE with the keycloak dep's domain + realm + client_secret + admin creds.
#
# This hook:
# 1. Reads the dep's connection info from $CCCI_DEPS_FILE.
# 2. Inserts the OIDC client secret as an abra app secret (recipe-conventional name oidc_rpcs).
# 3. Writes the OIDC env vars to the running app's .env via `abra app config set`.
# 4. Triggers an in-place `abra app deploy --force --chaos` so the new env takes effect.
# THIS IS NOT a fresh `abra app new` — the deploy-count guard (DG4.1, generalised) still
# sees one app_new per app.
#
# Env supplied by the orchestrator:
# CCCI_APP_DOMAIN — the running per-run lasuite-docs app domain
# CCCI_RECIPE — "lasuite-docs"
# CCCI_DEPS_FILE — JSON file (dict shape: {dep_recipe: {domain, realm, client_id, ...}, ...})
set -euo pipefail
: "${CCCI_APP_DOMAIN:?missing}"
: "${CCCI_DEPS_FILE:?missing}"
test -s "$CCCI_DEPS_FILE" || {
echo " setup_custom_tests: deps file empty"
exit 1
}
# Read keycloak dep info via jq
KC_DOMAIN=$(jq -r '.keycloak.domain' "$CCCI_DEPS_FILE")
KC_REALM=$(jq -r '.keycloak.realm' "$CCCI_DEPS_FILE")
KC_CLIENT=$(jq -r '.keycloak.client_id' "$CCCI_DEPS_FILE")
KC_SECRET=$(jq -r '.keycloak.client_secret' "$CCCI_DEPS_FILE")
if [ -z "$KC_DOMAIN" ] || [ "$KC_DOMAIN" = "null" ]; then
echo " setup_custom_tests: no keycloak.domain in deps"
exit 1
fi
if [ -z "$KC_SECRET" ] || [ "$KC_SECRET" = "null" ]; then
echo " setup_custom_tests: no keycloak.client_secret"
exit 1
fi
echo " lasuite-docs setup_custom_tests: wiring OIDC against keycloak dep ${KC_DOMAIN}"
# 1) Insert the OIDC client secret AT A BUMPED VERSION (the recipe-maintainer pattern).
# `abra app new -S` already generated `oidc_rpcs:v1` (random) — Docker Swarm forbids overwriting
# a secret at the same version, so we bump the version (v2), insert our value there, then
# update SECRET_OIDC_RPCS_VERSION in the .env to point at the new one.
ENV_PATH="$HOME/.abra/servers/default/${CCCI_APP_DOMAIN}.env"
CUR_VER=$(grep -E '^\s*SECRET_OIDC_RPCS_VERSION=' "$ENV_PATH" | tail -1 | cut -d= -f2 | tr -d '"\r' || echo "v1")
NEW_NUM=$((${CUR_VER#v} + 1))
NEW_VER="v${NEW_NUM}"
INSERT_LOG=$(abra app secret insert "$CCCI_APP_DOMAIN" oidc_rpcs "$NEW_VER" "$KC_SECRET" --no-input -C -o 2>&1) ||
INSERT_LOG=$(script -qec "abra app secret insert $CCCI_APP_DOMAIN oidc_rpcs $NEW_VER $KC_SECRET --no-input -C -o" /dev/null 2>&1) ||
{
echo " setup_custom_tests: abra app secret insert oidc_rpcs@$NEW_VER failed: $INSERT_LOG"
exit 1
}
# Repoint the env var to the new version
sed -i "s|^\s*SECRET_OIDC_RPCS_VERSION=.*|SECRET_OIDC_RPCS_VERSION=$NEW_VER|" "$ENV_PATH"
echo " setup_custom_tests: oidc_rpcs secret inserted at $NEW_VER (was $CUR_VER)"
# 2) Write OIDC env vars to the app's .env (names per lasuite-docs's .env.sample).
# Ensure the file ends with a newline FIRST so our appends don't concatenate onto the last line
# (we saw `TIMEOUT=900OIDC_REALM=...` malformed by a missing-trailing-newline file).
[ -z "$(tail -c1 "$ENV_PATH" 2>/dev/null)" ] || printf '\n' >>"$ENV_PATH"
write_env() {
local key="$1" val="$2"
# remove any existing key (commented or live) then append the live key=val
sed -i "/^\s*#\?\s*${key}=/d" "$ENV_PATH"
# Re-ensure trailing newline after each delete (sed may leave the file without one)
[ -z "$(tail -c1 "$ENV_PATH" 2>/dev/null)" ] || printf '\n' >>"$ENV_PATH"
printf '%s=%s\n' "$key" "$val" >>"$ENV_PATH"
}
write_env OIDC_REALM "$KC_REALM"
write_env OIDC_OP_DISCOVERY_ENDPOINT "https://${KC_DOMAIN}/realms/${KC_REALM}/.well-known/openid-configuration"
write_env OIDC_OP_AUTHORIZATION_ENDPOINT "https://${KC_DOMAIN}/realms/${KC_REALM}/protocol/openid-connect/auth"
write_env OIDC_OP_TOKEN_ENDPOINT "https://${KC_DOMAIN}/realms/${KC_REALM}/protocol/openid-connect/token"
write_env OIDC_OP_USER_ENDPOINT "https://${KC_DOMAIN}/realms/${KC_REALM}/protocol/openid-connect/userinfo"
write_env OIDC_OP_LOGOUT_ENDPOINT "https://${KC_DOMAIN}/realms/${KC_REALM}/protocol/openid-connect/logout"
write_env OIDC_OP_JWKS_ENDPOINT "https://${KC_DOMAIN}/realms/${KC_REALM}/protocol/openid-connect/certs"
write_env OIDC_RP_CLIENT_ID "$KC_CLIENT"
write_env OIDC_RP_SIGN_ALGO "RS256"
write_env OIDC_RP_SCOPES "openid email profile"
# 3) Trigger an in-place redeploy so the env update takes effect. --force re-deploys even when
# the recipe hasn't changed; --chaos avoids the chaos prompt; --no-input non-interactive.
abra app deploy "$CCCI_APP_DOMAIN" --force --chaos --no-input 2>&1 | tail -10
echo " lasuite-docs setup_custom_tests: OIDC wired + redeployed"

16
tests/lasuite-drive/functional/test_oidc_with_keycloak.py
View File

@ -3,12 +3,12 @@
Drive (La Suite Drive) is OIDC-required: login is gated by an external OpenID Connect provider. Drive (La Suite Drive) is OIDC-required: login is gated by an external OpenID Connect provider.
Mirrors the proven lasuite-docs SSO model: Mirrors the proven lasuite-docs SSO model:
- The orchestrator deploys a per-run keycloak dep AFTER the generic tiers and provisions a fresh - The orchestrator deploys a per-run keycloak dep AFTER the generic tiers and provisions a fresh
realm/client/user via `harness.sso.setup_keycloak_realm`; `setup_custom_tests.sh` then wires the realm/client/user via `harness.sso.setup_keycloak_realm`; `install_steps.sh` then wires the
OIDC env + client secret into the running drive app and redeploys. Creds land in `$CCCI_DEPS_FILE` OIDC env + client secret into the running drive app and redeploys. Creds land in `$CCCI_DEPS_FILE`
(read here via the `deps_creds` fixture). (read here via the `deps` fixture).
- This test consumes those creds and exercises the real OIDC flow against the dep keycloak: discovery - This test consumes those creds and exercises the real OIDC flow against the dep keycloak: discovery
endpoint advertises the realm, and a password grant yields a valid JWT with the expected claims. endpoint advertises the realm, and a password grant yields a valid JWT with the expected claims.
- Marked `@pytest.mark.requires_deps` so if setup_custom_tests failed the test SKIPs with a clear - Marked `@pytest.mark.requires_deps` so if dep provisioning failed the test SKIPs with a clear
`deps-not-ready` reason — and (per F2-11) the orchestrator then fails the run rather than going `deps-not-ready` reason — and (per F2-11) the orchestrator then fails the run rather than going
green on a skipped SSO test. green on a skipped SSO test.
@ -36,13 +36,13 @@ def _b64url_decode(seg: str) -> bytes:
@pytest.mark.requires_deps @pytest.mark.requires_deps
def test_oidc_password_grant_against_dep_keycloak(live_app, deps_creds): def test_oidc_password_grant_against_dep_keycloak(live_app, deps):
"""The dep keycloak issues a JWT for the pre-provisioned test user via OIDC password grant.""" """The dep keycloak issues a JWT for the pre-provisioned test user via OIDC password grant."""
assert "keycloak" in deps_creds, ( assert "keycloak" in deps, (
f"keycloak creds not in deps_creds; got {list(deps_creds.keys())}. " f"keycloak creds not in deps; got {list(deps.keys())}. "
"setup_custom_tests should have populated this." "dep provisioning should have populated this."
) )
kc = deps_creds["keycloak"] kc = deps["keycloak"]
# Creds shape. WC1: realm is per-run namespaced "<parent>-<6hex>"; client_id stays the parent. # Creds shape. WC1: realm is per-run namespaced "<parent>-<6hex>"; client_id stays the parent.
assert kc["domain"] assert kc["domain"]

2
tests/lasuite-drive/install_steps.sh
View File

@ -6,7 +6,7 @@
# BEFORE the single `abra app deploy` (runner/harness/lifecycle.py::_run_install_steps). By writing # BEFORE the single `abra app deploy` (runner/harness/lifecycle.py::_run_install_steps). By writing
# the OIDC env + the real client secret into the app's `.env` HERE, the recipe deploys ONCE with # the OIDC env + the real client secret into the app's `.env` HERE, the recipe deploys ONCE with
# OIDC already wired — eliminating the flaky post-deploy in-place `--force --chaos` 12-service # OIDC already wired — eliminating the flaky post-deploy in-place `--force --chaos` 12-service
# reconverge that the old setup_custom_tests.sh did (collabora WOPI-discovery race; see JOURNAL # post-deploy reconverge (collabora WOPI-discovery race; see JOURNAL
# Step 0). The orchestrator provisions the per-run realm/client on the live-warm keycloak BEFORE # Step 0). The orchestrator provisions the per-run realm/client on the live-warm keycloak BEFORE
# this hook and writes $CCCI_DEPS_FILE (the recipe→creds dict). # this hook and writes $CCCI_DEPS_FILE (the recipe→creds dict).
# #

58
tests/lasuite-drive/ops.py
View File

@ -5,6 +5,7 @@ in the `db` service. The backup path exercises the recipe's pg_backup.sh DB-dump
backupbot-labelled).""" backupbot-labelled)."""
import os import os
import subprocess
import sys import sys
import time import time
@ -12,6 +13,47 @@ sys.path.insert(0, os.path.join(os.path.dirname(__file__), "..", "..", "runner")
from harness import lifecycle # noqa: E402 from harness import lifecycle # noqa: E402
def pre_install(ctx):
"""Post-deploy seed for the custom tier (the former setup_custom_tests.sh, moved here in rcust
P2b — install_steps.sh runs PRE-deploy and cannot touch the live stack). The deploy alone does
NOT create the MinIO bucket: `minio-createbuckets` is a `replicas:0` one-shot (restart_policy:
none) that must be triggered. The MinIO storage test asserts the bucket exists, so trigger it
here and poll. `--detach` is REQUIRED: the job creates the bucket then EXITS 0, so it never
holds a steady 1/1 replica — a blocking scale would wait forever."""
stack = ctx.domain.replace(".", "_")
print(" pre_install: creating MinIO bucket via the minio-createbuckets one-shot", flush=True)
subprocess.run(
["docker", "service", "scale", "--detach", f"{stack}_minio-createbuckets=1"],
capture_output=True,
check=False,
)
check = (
'mc alias set _c http://localhost:9000 "$(cat /run/secrets/minio_ru)" '
'"$(cat /run/secrets/minio_rp)" >/dev/null 2>&1 && '
"mc ls _c/drive-media-storage >/dev/null 2>&1"
)
for i in range(30):
cid = subprocess.run(
["docker", "ps", "-q", "-f", f"name={stack}_minio.1"],
capture_output=True,
text=True,
check=False,
).stdout.split()
if cid and (
subprocess.run(
["docker", "exec", cid[0], "sh", "-c", check], capture_output=True, check=False
).returncode
== 0
):
print(
f" pre_install: bucket drive-media-storage present after {i + 1} poll(s)",
flush=True,
)
return
time.sleep(3)
raise AssertionError("minio-createbuckets one-shot did not create drive-media-storage in 90s")
def _wait_collabora_ready(domain, timeout=420): def _wait_collabora_ready(domain, timeout=420):
"""Gate the upgrade op on collabora being FULLY ready (WOPI discovery endpoint → 200), not just """Gate the upgrade op on collabora being FULLY ready (WOPI discovery endpoint → 200), not just
container 1/1 'running'. coolwsd takes ~2min to boot (pre-reads 1300+ l10n files + RSA keygen); container 1/1 'running'. coolwsd takes ~2min to boot (pre-reads 1300+ l10n files + RSA keygen);
@ -49,21 +91,21 @@ def _seed(domain, value):
assert _psql(domain, "SELECT v FROM ci_marker;") == value assert _psql(domain, "SELECT v FROM ci_marker;") == value
def pre_upgrade(domain, meta): def pre_upgrade(ctx):
# Gate the chaos redeploy on a fully-ready collabora (else it kills a still-booting coolwsd and # Gate the chaos redeploy on a fully-ready collabora (else it kills a still-booting coolwsd and
# abra aborts the upgrade deploy — Q3.2a run 1). Then seed the data-integrity marker. # abra aborts the upgrade deploy — Q3.2a run 1). Then seed the data-integrity marker.
_wait_collabora_ready(domain) _wait_collabora_ready(ctx.domain)
_seed(domain, "upgrade-survives") _seed(ctx.domain, "upgrade-survives")
def pre_backup(domain, meta): def pre_backup(ctx):
_seed(domain, "original") _seed(ctx.domain, "original")
def pre_restore(domain, meta): def pre_restore(ctx):
# drop the marker table (diverge from the backup) so a successful restore is observable # drop the marker table (diverge from the backup) so a successful restore is observable
_psql(domain, "DROP TABLE ci_marker;") _psql(ctx.domain, "DROP TABLE ci_marker;")
assert _psql(domain, "SELECT to_regclass('public.ci_marker');") in ( assert _psql(ctx.domain, "SELECT to_regclass('public.ci_marker');") in (
"", "",
"NULL", "NULL",
), "drop did not take" ), "drop did not take"

33
tests/lasuite-drive/recipe_meta.py
View File

@ -18,34 +18,31 @@ DEPLOY_TIMEOUT = 1800
HTTP_TIMEOUT = 900 HTTP_TIMEOUT = 900
# Base deploy/lifecycle proven cold-green @2026-05-28 (install: pass; 12 services incl. # Base deploy/lifecycle proven cold-green @2026-05-28 (install: pass; 12 services incl.
# onlyoffice+collabora) once the Docker Hub rate limit was fixed. The keycloak SSO dep is now # onlyoffice+collabora) once the Docker Hub rate limit was fixed. Declaring DEPS makes the
# enabled: declaring DEPS triggers the orchestrator's setup_custom_tests step (deploy keycloak + # orchestrator provision keycloak (realm/client/user) BEFORE the single deploy;
# provision realm/client/user + run tests/lasuite-drive/setup_custom_tests.sh to wire OIDC env + # functional/test_oidc_with_keycloak.py then exercises the SSO flow.
# in-place redeploy). functional/test_oidc_with_keycloak.py then exercises the SSO flow.
DEPS = ["keycloak"] DEPS = ["keycloak"]
# Q3.2a (plan-lasuite-drive-oidc-robustness.md Part A): wire OIDC at INSTALL time, not via a # OIDC is wired at INSTALL time (the only deps mode since rcust P2b; Q3.2a pioneered it here):
# post-deploy in-place `--chaos` redeploy. The orchestrator provisions the per-run realm on the # the orchestrator provisions the per-run realm on the live-warm keycloak BEFORE the single
# live-warm keycloak BEFORE the single `abra app deploy`, and tests/lasuite-drive/install_steps.sh # `abra app deploy`, and tests/lasuite-drive/install_steps.sh writes the OIDC env + client secret
# writes the OIDC env + client secret into the .env that one deploy reads. This eliminates the flaky # into the .env that one deploy reads. No post-deploy reconverge (the flaky 12-service collabora
# 12-service reconverge (collabora WOPI-discovery race; JOURNAL Step 0). Drive boots fine with OIDC # WOPI race is structurally gone). The post-deploy MinIO bucket one-shot lives in ops.py
# env set because keycloak is live-warm (discovery reachable at boot). setup_custom_tests.sh now # pre_install (the former setup_custom_tests.sh, deleted in P2b).
# only triggers the post-deploy MinIO bucket one-shot.
OIDC_AT_INSTALL = True
def READY_PROBE(domain): def READY_PROBE(ctx):
"""Readiness signals beyond replica-convergence + the app HEALTH_PATH (Q3.2/F2-12). collabora's """Readiness signals beyond replica-convergence + the app HEALTH_PATH (Q3.2/F2-12). collabora's
coolwsd reports its container 1/1 'running' while still doing jail/config init, and its WOPI coolwsd reports its container 1/1 'running' while still doing jail/config init, and its WOPI
discovery endpoint 404s until ready — so the harness waits for `/hosting/discovery` → 200 on the discovery endpoint 404s until ready — so the harness waits for `/hosting/discovery` → 200 on the
collabora sibling host after the install deploy AND after the upgrade chaos redeploy. This is what collabora sibling host after the install deploy AND after the upgrade chaos redeploy. This is what
makes the heavy prev→PR-head crossover reliably green (the new collabora 25.04.9.x finishes init makes the heavy prev→PR-head crossover reliably green (the new collabora 25.04.9.x finishes init
within swarm's healthcheck retries; abra's own converge monitor was too impatient — F2-12).""" within swarm's healthcheck retries; abra's own converge monitor was too impatient — F2-12)."""
label, _, rest = domain.partition(".") label, _, rest = ctx.domain.partition(".")
return [{"host": f"collabora-{domain}", "path": "/hosting/discovery", "ok": (200,)}] return [{"host": f"collabora-{ctx.domain}", "path": "/hosting/discovery", "ok": (200,)}]
def EXTRA_ENV(domain): def EXTRA_ENV(ctx):
# Two of lasuite-drive's services route on DOMAIN-DERIVED **nested** subdomains — # Two of lasuite-drive's services route on DOMAIN-DERIVED **nested** subdomains —
# `MINIO_DOMAIN="minio.${DOMAIN}"` and `COLLABORA_DOMAIN="collabora.${DOMAIN}"`. The cc-ci # `MINIO_DOMAIN="minio.${DOMAIN}"` and `COLLABORA_DOMAIN="collabora.${DOMAIN}"`. The cc-ci
# wildcard TLS cert is `*.ci.commoninternet.net` (single label only), so a 2-label name like # wildcard TLS cert is `*.ci.commoninternet.net` (single label only), so a 2-label name like
@ -55,8 +52,8 @@ def EXTRA_ENV(domain):
# no cert/gateway change. See DECISIONS.md "Phase 2 — nested DOMAIN-derived subdomains". # no cert/gateway change. See DECISIONS.md "Phase 2 — nested DOMAIN-derived subdomains".
# `AWS_S3_DOMAIN_REPLACE` derives from MINIO_DOMAIN in-compose, so setting MINIO_DOMAIN is enough. # `AWS_S3_DOMAIN_REPLACE` derives from MINIO_DOMAIN in-compose, so setting MINIO_DOMAIN is enough.
return { return {
"MINIO_DOMAIN": f"minio-{domain}", "MINIO_DOMAIN": f"minio-{ctx.domain}",
"COLLABORA_DOMAIN": f"collabora-{domain}", "COLLABORA_DOMAIN": f"collabora-{ctx.domain}",
# abra's internal per-deploy convergence timeout (recipe TIMEOUT env, default 300s) is too # abra's internal per-deploy convergence timeout (recipe TIMEOUT env, default 300s) is too
# short for this 12-service stack on a cold image cache (impress frontend/backend, minio, # short for this 12-service stack on a cold image cache (impress frontend/backend, minio,
# postgres, redis, collabora ~1GB, onlyoffice ~2GB). Bump so abra waits long enough for # postgres, redis, collabora ~1GB, onlyoffice ~2GB). Bump so abra waits long enough for

39
tests/lasuite-drive/setup_custom_tests.sh
View File

@ -1,39 +0,0 @@
#!/usr/bin/env bash
# lasuite-drive — POST-DEPLOY setup hook (Phase 2 Q3.2a).
#
# As of Q3.2a (plan-lasuite-drive-oidc-robustness.md Part A) OIDC is wired at INSTALL time by
# tests/lasuite-drive/install_steps.sh (before the single `abra app deploy`), so this hook NO LONGER
# does any OIDC env wiring or in-place redeploy — that eliminated the flaky 12-service reconverge
# (collabora WOPI race; see JOURNAL Step 0). What remains here is the ONE post-deploy step that
# genuinely needs the live stack: triggering the MinIO bucket-creation one-shot. The orchestrator
# runs this only on the install-time path AFTER the deploy is healthy (deps already provisioned).
#
# Env supplied by the orchestrator:
# CCCI_APP_DOMAIN — the running per-run lasuite-drive app domain
# CCCI_DEPS_FILE — JSON deps creds dict (unused here now; OIDC handled at install)
set -euo pipefail
: "${CCCI_APP_DOMAIN:?missing}"
# The deploy alone does NOT create the MinIO bucket — `minio-createbuckets` is a `replicas:0`
# one-shot (restart_policy: none) that must be triggered. The MinIO storage test asserts the bucket
# exists, so create it here. `--detach` is REQUIRED: the job creates the bucket then EXITS 0, so it
# never holds a steady 1/1 replica; a blocking `docker service scale ...=1` would wait forever and
# hang the run. With `--detach` the scale just submits the one-run and returns; the poll loop below
# confirms the bucket was actually created.
STACK=$(printf '%s' "$CCCI_APP_DOMAIN" | tr '.' '_')
echo " setup: creating MinIO bucket via the minio-createbuckets one-shot (scale 0->1)"
docker service scale --detach "${STACK}_minio-createbuckets=1" >/dev/null 2>&1 || true
# Wait up to 90s for the one-shot to create the bucket (mc mb drive/drive-media-storage; exit 0).
# Poll by checking the bucket directly from the running minio replica container.
for i in $(seq 1 30); do
MC_CID=$(docker ps -q -f "name=${STACK}_minio.1" | head -1)
if [ -n "$MC_CID" ] && docker exec "$MC_CID" sh -c \
'mc alias set _c http://localhost:9000 "$(cat /run/secrets/minio_ru)" "$(cat /run/secrets/minio_rp)" >/dev/null 2>&1 && mc ls _c/drive-media-storage >/dev/null 2>&1'; then
echo " setup: bucket drive-media-storage present after ${i} poll(s)"
break
fi
sleep 3
done
echo " lasuite-drive setup_custom_tests: post-deploy MinIO bucket step complete (OIDC wired at install)"

10
tests/lasuite-meet/functional/test_meeting_flow.py
View File

@ -36,8 +36,8 @@ def _b64url(seg: str) -> bytes:
return base64.urlsafe_b64decode(seg + "=" * ((4 - len(seg) % 4) % 4)) return base64.urlsafe_b64decode(seg + "=" * ((4 - len(seg) % 4) % 4))
def _creds(deps_creds: dict) -> dict: def _creds(deps: dict) -> dict:
kc = deps_creds["keycloak"] kc = deps["keycloak"]
return { return {
"provider": "keycloak", "provider": "keycloak",
"provider_domain": kc["domain"], "provider_domain": kc["domain"],
@ -55,10 +55,10 @@ def _creds(deps_creds: dict) -> dict:
@pytest.mark.requires_deps @pytest.mark.requires_deps
def test_create_room_get_livekit_token_and_read_back(live_app, deps_creds): def test_create_room_get_livekit_token_and_read_back(live_app, deps):
assert "keycloak" in deps_creds, f"keycloak creds missing; got {list(deps_creds.keys())}" assert "keycloak" in deps, f"keycloak creds missing; got {list(deps.keys())}"
base = f"https://{live_app}" base = f"https://{live_app}"
token = sso.oidc_password_grant(_creds(deps_creds)) token = sso.oidc_password_grant(_creds(deps))
assert isinstance(token, str) and token.count(".") == 2, "OIDC access token is not a JWT" assert isinstance(token, str) and token.count(".") == 2, "OIDC access token is not a JWT"
auth = {"Authorization": f"Bearer {token}"} auth = {"Authorization": f"Bearer {token}"}

16
tests/lasuite-meet/functional/test_oidc_with_keycloak.py
View File

@ -3,12 +3,12 @@
Meet (La Suite Meet) is OIDC-required: login is gated by an external OpenID Connect provider. Meet (La Suite Meet) is OIDC-required: login is gated by an external OpenID Connect provider.
Mirrors the proven lasuite-docs SSO model: Mirrors the proven lasuite-docs SSO model:
- The orchestrator deploys a per-run keycloak dep AFTER the generic tiers and provisions a fresh - The orchestrator deploys a per-run keycloak dep AFTER the generic tiers and provisions a fresh
realm/client/user via `harness.sso.setup_keycloak_realm`; `setup_custom_tests.sh` then wires the realm/client/user via `harness.sso.setup_keycloak_realm`; `install_steps.sh` then wires the
OIDC env + client secret into the running drive app and redeploys. Creds land in `$CCCI_DEPS_FILE` OIDC env + client secret into the running drive app and redeploys. Creds land in `$CCCI_DEPS_FILE`
(read here via the `deps_creds` fixture). (read here via the `deps` fixture).
- This test consumes those creds and exercises the real OIDC flow against the dep keycloak: discovery - This test consumes those creds and exercises the real OIDC flow against the dep keycloak: discovery
endpoint advertises the realm, and a password grant yields a valid JWT with the expected claims. endpoint advertises the realm, and a password grant yields a valid JWT with the expected claims.
- Marked `@pytest.mark.requires_deps` so if setup_custom_tests failed the test SKIPs with a clear - Marked `@pytest.mark.requires_deps` so if dep provisioning failed the test SKIPs with a clear
`deps-not-ready` reason — and (per F2-11) the orchestrator then fails the run rather than going `deps-not-ready` reason — and (per F2-11) the orchestrator then fails the run rather than going
green on a skipped SSO test. green on a skipped SSO test.
@ -36,13 +36,13 @@ def _b64url_decode(seg: str) -> bytes:
@pytest.mark.requires_deps @pytest.mark.requires_deps
def test_oidc_password_grant_against_dep_keycloak(live_app, deps_creds): def test_oidc_password_grant_against_dep_keycloak(live_app, deps):
"""The dep keycloak issues a JWT for the pre-provisioned test user via OIDC password grant.""" """The dep keycloak issues a JWT for the pre-provisioned test user via OIDC password grant."""
assert "keycloak" in deps_creds, ( assert "keycloak" in deps, (
f"keycloak creds not in deps_creds; got {list(deps_creds.keys())}. " f"keycloak creds not in deps; got {list(deps.keys())}. "
"setup_custom_tests should have populated this." "dep provisioning should have populated this."
) )
kc = deps_creds["keycloak"] kc = deps["keycloak"]
# Creds shape. WC1: realm is per-run namespaced "<parent>-<6hex>"; client_id stays the parent. # Creds shape. WC1: realm is per-run namespaced "<parent>-<6hex>"; client_id stays the parent.
assert kc["domain"] assert kc["domain"]

3
tests/lasuite-meet/install_steps.sh
View File

@ -4,7 +4,8 @@
# Runs during the install tier AFTER `abra app new` + EXTRA_ENV + `abra app secret generate`, and # Runs during the install tier AFTER `abra app new` + EXTRA_ENV + `abra app secret generate`, and
# BEFORE the single `abra app deploy` (lifecycle.py::_run_install_steps). Writing OIDC env + the real # BEFORE the single `abra app deploy` (lifecycle.py::_run_install_steps). Writing OIDC env + the real
# client secret HERE means the recipe deploys ONCE with OIDC already wired — no post-deploy reconverge # client secret HERE means the recipe deploys ONCE with OIDC already wired — no post-deploy reconverge
# (OIDC_AT_INSTALL). The orchestrator provisions the per-run realm/client on the live-warm keycloak # (install-time deps wiring — the only mode since rcust P2b). The orchestrator provisions the
# per-run realm/client on the live-warm keycloak
# BEFORE this hook and writes $CCCI_DEPS_FILE (the recipe→creds dict). # BEFORE this hook and writes $CCCI_DEPS_FILE (the recipe→creds dict).
# #
# Meet's OIDC is REQUIRED (recipe README). Same La Suite/impress env contract as drive, with meet's # Meet's OIDC is REQUIRED (recipe README). Same La Suite/impress env contract as drive, with meet's

14
tests/lasuite-meet/ops.py
View File

@ -27,18 +27,18 @@ def _seed(domain, value):
assert _psql(domain, "SELECT v FROM ci_marker;") == value assert _psql(domain, "SELECT v FROM ci_marker;") == value
def pre_upgrade(domain, meta): def pre_upgrade(ctx):
_seed(domain, "upgrade-survives") _seed(ctx.domain, "upgrade-survives")
def pre_backup(domain, meta): def pre_backup(ctx):
_seed(domain, "original") _seed(ctx.domain, "original")
def pre_restore(domain, meta): def pre_restore(ctx):
# drop the marker table (diverge from the backup) so a successful restore is observable # drop the marker table (diverge from the backup) so a successful restore is observable
_psql(domain, "DROP TABLE ci_marker;") _psql(ctx.domain, "DROP TABLE ci_marker;")
assert _psql(domain, "SELECT to_regclass('public.ci_marker');") in ( assert _psql(ctx.domain, "SELECT to_regclass('public.ci_marker');") in (
"", "",
"NULL", "NULL",
), "drop did not take" ), "drop did not take"

15
tests/lasuite-meet/recipe_meta.py
View File

@ -13,16 +13,15 @@ HEALTH_OK = (200, 301, 302)
DEPLOY_TIMEOUT = 1200 DEPLOY_TIMEOUT = 1200
HTTP_TIMEOUT = 600 HTTP_TIMEOUT = 600
# SSO-dependent (recipe.toml requires=["keycloak"], [sso] provider=keycloak). Wire OIDC at INSTALL # SSO-dependent (recipe.toml requires=["keycloak"], [sso] provider=keycloak). OIDC is wired at
# time against the live-warm keycloak — same machinery as lasuite-drive (Q3.2a): the orchestrator # INSTALL time (the only deps mode since rcust P2b) against the live-warm keycloak: the
# provisions the per-run realm BEFORE the single `abra app deploy`, and tests/lasuite-meet/ # orchestrator provisions the per-run realm BEFORE the single `abra app deploy`, and
# install_steps.sh writes the OIDC env + client secret into that one deploy (no post-deploy # tests/lasuite-meet/install_steps.sh writes the OIDC env + client secret into that one deploy
# reconverge). Meet boots fine with OIDC env set because keycloak is live-warm. # (no post-deploy reconverge). Meet boots fine with OIDC env set because keycloak is live-warm.
DEPS = ["keycloak"] DEPS = ["keycloak"]
OIDC_AT_INSTALL = True
def EXTRA_ENV(domain): def EXTRA_ENV(ctx):
# lasuite-meet routes LiveKit's WebSocket signaling on a DOMAIN-derived **nested** subdomain # lasuite-meet routes LiveKit's WebSocket signaling on a DOMAIN-derived **nested** subdomain
# `LIVEKIT_DOMAIN="livekit.${DOMAIN}"`. The cc-ci wildcard TLS cert is `*.ci.commoninternet.net` # `LIVEKIT_DOMAIN="livekit.${DOMAIN}"`. The cc-ci wildcard TLS cert is `*.ci.commoninternet.net`
# (single label only), so a 2-label name like `livekit.lasuite-meet-pr0-abc.ci.commoninternet.net` # (single label only), so a 2-label name like `livekit.lasuite-meet-pr0-abc.ci.commoninternet.net`
@ -31,7 +30,7 @@ def EXTRA_ENV(domain):
# no cert/gateway change. Same fix as lasuite-drive's minio/collabora siblings (DECISIONS.md # no cert/gateway change. Same fix as lasuite-drive's minio/collabora siblings (DECISIONS.md
# "Phase 2 — nested DOMAIN-derived subdomains"). # "Phase 2 — nested DOMAIN-derived subdomains").
return { return {
"LIVEKIT_DOMAIN": f"livekit-{domain}", "LIVEKIT_DOMAIN": f"livekit-{ctx.domain}",
# abra's internal per-deploy convergence TIMEOUT (default 300s) is too short for this stack on # abra's internal per-deploy convergence TIMEOUT (default 300s) is too short for this stack on
# a cold image cache; bump it (kept under DEPLOY_TIMEOUT so Python never kills abra mid-wait). # a cold image cache; bump it (kept under DEPLOY_TIMEOUT so Python never kills abra mid-wait).
"TIMEOUT": "1000", "TIMEOUT": "1000",

6
tests/mailu/recipe_meta.py
View File

@ -21,10 +21,10 @@ DEPLOY_TIMEOUT = 900
HTTP_TIMEOUT = 600 HTTP_TIMEOUT = 600
def EXTRA_ENV(domain): def EXTRA_ENV(ctx):
return { return {
"MAIL_DOMAIN": domain, "MAIL_DOMAIN": ctx.domain,
"HOSTNAMES": domain, "HOSTNAMES": ctx.domain,
"TRAEFIK_STACK_NAME": "traefik_ci_commoninternet_net", "TRAEFIK_STACK_NAME": "traefik_ci_commoninternet_net",
"TLS_FLAVOR": "notls", "TLS_FLAVOR": "notls",
"SITENAME": "ccci-mail", "SITENAME": "ccci-mail",

14
tests/matrix-synapse/ops.py
View File

@ -24,18 +24,18 @@ def _seed(domain, value):
assert _psql(domain, "SELECT v FROM ci_marker;") == value assert _psql(domain, "SELECT v FROM ci_marker;") == value
def pre_upgrade(domain, meta): def pre_upgrade(ctx):
_seed(domain, "upgrade-survives") _seed(ctx.domain, "upgrade-survives")
def pre_backup(domain, meta): def pre_backup(ctx):
_seed(domain, "original") _seed(ctx.domain, "original")
def pre_restore(domain, meta): def pre_restore(ctx):
# drop the marker table (diverge from the backup) so a successful restore is observable # drop the marker table (diverge from the backup) so a successful restore is observable
_psql(domain, "DROP TABLE ci_marker;") _psql(ctx.domain, "DROP TABLE ci_marker;")
assert _psql(domain, "SELECT to_regclass('public.ci_marker');") in ( assert _psql(ctx.domain, "SELECT to_regclass('public.ci_marker');") in (
"", "",
"NULL", "NULL",
), "drop did not take" ), "drop did not take"

14
tests/mattermost-lts/ops.py
View File

@ -29,18 +29,18 @@ def _seed(domain, value):
assert _psql(domain, "SELECT v FROM ci_marker;") == value assert _psql(domain, "SELECT v FROM ci_marker;") == value
def pre_upgrade(domain, meta): def pre_upgrade(ctx):
_seed(domain, "upgrade-survives") _seed(ctx.domain, "upgrade-survives")
def pre_backup(domain, meta): def pre_backup(ctx):
_seed(domain, "original") _seed(ctx.domain, "original")
def pre_restore(domain, meta): def pre_restore(ctx):
# drop the marker table (diverge from the backup) so a successful restore is observable # drop the marker table (diverge from the backup) so a successful restore is observable
_psql(domain, "DROP TABLE ci_marker;") _psql(ctx.domain, "DROP TABLE ci_marker;")
assert _psql(domain, "SELECT to_regclass('public.ci_marker');") in ( assert _psql(ctx.domain, "SELECT to_regclass('public.ci_marker');") in (
"", "",
"NULL", "NULL",
), "drop did not take" ), "drop did not take"

4
tests/mumble/functional/test_server_config_limits.py
View File

@ -26,9 +26,9 @@ def test_configured_max_users_surfaces_in_serverconfig(live_app):
assert r["server_sync"], f"ServerSync handshake did not complete — {r.get('error')}" assert r["server_sync"], f"ServerSync handshake did not complete — {r.get('error')}"
cfg = r["server_config"] cfg = r["server_config"]
assert cfg, f"server did not send a ServerConfig message — {r!r}" assert cfg, f"server did not send a ServerConfig message — {r!r}"
assert cfg.get("max_users") == recipe_meta.MAX_USERS, ( assert cfg.get("max_users") == recipe_meta._MAX_USERS, (
f"ServerConfig.max_users={cfg.get('max_users')!r} does not match the configured " f"ServerConfig.max_users={cfg.get('max_users')!r} does not match the configured "
f"USERS={recipe_meta.MAX_USERS} — deploy-time server-limit config did not propagate" f"USERS={recipe_meta._MAX_USERS} — deploy-time server-limit config did not propagate"
) )
# allow_html defaults true in the recipe; assert it is present/boolean to prove the field set # allow_html defaults true in the recipe; assert it is present/boolean to prove the field set
# is the real ServerConfig (not an empty/garbled decode). # is the real ServerConfig (not an empty/garbled decode).

2
tests/mumble/functional/test_welcome_text_roundtrip.py
View File

@ -20,7 +20,7 @@ import recipe_meta # noqa: E402
def test_configured_welcome_text_surfaces_in_serversync(live_app): def test_configured_welcome_text_surfaces_in_serversync(live_app):
marker = recipe_meta.WELCOME_TEXT_MARKER marker = recipe_meta._WELCOME_TEXT_MARKER
r = _mumble_proto.retry_handshake(attempts=12, interval=5.0) r = _mumble_proto.retry_handshake(attempts=12, interval=5.0)
assert r["server_sync"], f"ServerSync handshake did not complete — {r.get('error')}" assert r["server_sync"], f"ServerSync handshake did not complete — {r.get('error')}"

16
tests/mumble/ops.py
View File

@ -38,16 +38,18 @@ def _seed(domain, value):
assert got == value, f"seed did not commit (read back {got!r}, expected {value!r})" assert got == value, f"seed did not commit (read back {got!r}, expected {value!r})"
def pre_upgrade(domain, meta): def pre_upgrade(ctx):
_seed(domain, "upgrade-survives") _seed(ctx.domain, "upgrade-survives")
def pre_backup(domain, meta): def pre_backup(ctx):
_seed(domain, "original") _seed(ctx.domain, "original")
def pre_restore(domain, meta): def pre_restore(ctx):
# diverge from the backup so a successful restore is observable: drop the marker table. # diverge from the backup so a successful restore is observable: drop the marker table.
_sqlite(domain, "DROP TABLE IF EXISTS ci_marker;") _sqlite(ctx.domain, "DROP TABLE IF EXISTS ci_marker;")
got = _sqlite(domain, "SELECT name FROM sqlite_master WHERE type='table' AND name='ci_marker';") got = _sqlite(
ctx.domain, "SELECT name FROM sqlite_master WHERE type='table' AND name='ci_marker';"
)
assert got == "", f"drop did not take (sqlite_master still lists ci_marker: {got!r})" assert got == "", f"drop did not take (sqlite_master still lists ci_marker: {got!r})"

15
tests/mumble/recipe_meta.py
View File

@ -31,18 +31,19 @@ HEALTH_OK = (200,)
DEPLOY_TIMEOUT = 900 # two images to pull (mumble-server + mumble-web) on a cold node DEPLOY_TIMEOUT = 900 # two images to pull (mumble-server + mumble-web) on a cold node
HTTP_TIMEOUT = 300 HTTP_TIMEOUT = 300
# A unique, stable welcome-text marker the round-trip test asserts surfaces over the protocol. # A unique, stable welcome-text marker the round-trip test asserts surfaces over the protocol
WELCOME_TEXT_MARKER = "cc-ci-mumble-welcome-7f3a9c" # (underscore prefix = recipe-private constant, exempt from registry validation — rcust P1).
_WELCOME_TEXT_MARKER = "cc-ci-mumble-welcome-7f3a9c"
# A distinctive max-users value (not the recipe default 100) the server_config test asserts. # A distinctive max-users value (not the recipe default 100) the server_config test asserts.
MAX_USERS = 42 _MAX_USERS = 42
# BASE deploy (0.2.0): mumble-web only — NO host-ports (0.2.0 predates it). The voice-config env is # BASE deploy (0.2.0): mumble-web only — NO host-ports (0.2.0 predates it). The voice-config env is
# set here and persists across the upgrade so it takes effect on the latest (where the custom config # set here and persists across the upgrade so it takes effect on the latest (where the custom config
# round-trip tests assert it). # round-trip tests assert it).
EXTRA_ENV = { EXTRA_ENV = {
"COMPOSE_FILE": "compose.yml:compose.mumbleweb.yml", "COMPOSE_FILE": "compose.yml:compose.mumbleweb.yml",
"WELCOME_TEXT": WELCOME_TEXT_MARKER, "WELCOME_TEXT": _WELCOME_TEXT_MARKER,
"USERS": str(MAX_USERS), "USERS": str(_MAX_USERS),
} }
# UPGRADE-target deploy (latest 1.0.0+): add the NATIVE compose.host-ports.yml so 64738 is # UPGRADE-target deploy (latest 1.0.0+): add the NATIVE compose.host-ports.yml so 64738 is
@ -52,7 +53,7 @@ UPGRADE_EXTRA_ENV = {
} }
def READY_PROBE(domain): def READY_PROBE(ctx):
# The voice server on 64738 is testable on-host ONLY when compose.host-ports.yml is active — i.e. # The voice server on 64738 is testable on-host ONLY when compose.host-ports.yml is active — i.e.
# the post-upgrade LATEST, not the minimal 0.2.0 base. Read the live COMPOSE_FILE to decide, so the # the post-upgrade LATEST, not the minimal 0.2.0 base. Read the live COMPOSE_FILE to decide, so the
# SAME probe fn is correct in both phases: the post-install probe (base, no host-ports) returns [] # SAME probe fn is correct in both phases: the post-install probe (base, no host-ports) returns []
@ -63,7 +64,7 @@ def READY_PROBE(domain):
# backup-bot would then exec into a not-running app container -> 409). # backup-bot would then exec into a not-running app container -> 409).
from harness import abra # lazy: recipe_meta is exec'd with `harness` importable at call time from harness import abra # lazy: recipe_meta is exec'd with `harness` importable at call time
cf = abra.env_get(domain, "COMPOSE_FILE") or "" cf = abra.env_get(ctx.domain, "COMPOSE_FILE") or ""
if "compose.host-ports.yml" in cf: if "compose.host-ports.yml" in cf:
return [{"tcp_host": "127.0.0.1", "tcp_port": 64738, "stable": 3}] return [{"tcp_host": "127.0.0.1", "tcp_port": 64738, "stable": 3}]
return [] return []

12
tests/n8n/ops.py
View File

@ -15,13 +15,13 @@ def _write(domain, val):
lifecycle.exec_in_app(domain, ["sh", "-c", f"echo {val} > {MARKER}"]) lifecycle.exec_in_app(domain, ["sh", "-c", f"echo {val} > {MARKER}"])
def pre_upgrade(domain, meta): def pre_upgrade(ctx):
_write(domain, "upgrade-survives") _write(ctx.domain, "upgrade-survives")
def pre_backup(domain, meta): def pre_backup(ctx):
_write(domain, "original") _write(ctx.domain, "original")
def pre_restore(domain, meta): def pre_restore(ctx):
_write(domain, "mutated") # diverge so a successful restore is observable _write(ctx.domain, "mutated") # diverge so a successful restore is observable

14
tests/plausible/ops.py
View File

@ -24,17 +24,17 @@ def _seed(domain, value):
assert _psql(domain, "SELECT v FROM ci_marker;") == value assert _psql(domain, "SELECT v FROM ci_marker;") == value
def pre_upgrade(domain, meta): def pre_upgrade(ctx):
_seed(domain, "upgrade-survives") _seed(ctx.domain, "upgrade-survives")
def pre_backup(domain, meta): def pre_backup(ctx):
_seed(domain, "original") _seed(ctx.domain, "original")
def pre_restore(domain, meta): def pre_restore(ctx):
_psql(domain, "DROP TABLE ci_marker;") _psql(ctx.domain, "DROP TABLE ci_marker;")
assert _psql(domain, "SELECT to_regclass('public.ci_marker');") in ( assert _psql(ctx.domain, "SELECT to_regclass('public.ci_marker');") in (
"", "",
"NULL", "NULL",
), "drop did not take" ), "drop did not take"

13
tests/unit/test_canonical.py
View File

@ -13,6 +13,7 @@ import sys
sys.path.insert(0, os.path.join(os.path.dirname(__file__), "..", "..", "runner")) sys.path.insert(0, os.path.join(os.path.dirname(__file__), "..", "..", "runner"))
from harness import canonical, warm # noqa: E402 from harness import canonical, warm # noqa: E402
from harness import meta as harness_meta # noqa: E402
def test_canonical_domain(): def test_canonical_domain():
@ -33,11 +34,9 @@ def test_is_enrolled_reads_flag(tmp_path, monkeypatch):
tests_dir = tmp_path / "tests" / recipe tests_dir = tmp_path / "tests" / recipe
tests_dir.mkdir(parents=True) tests_dir.mkdir(parents=True)
(tests_dir / "recipe_meta.py").write_text("WARM_CANONICAL = True\n") (tests_dir / "recipe_meta.py").write_text("WARM_CANONICAL = True\n")
# canonical.is_enrolled builds the path from canonical.__file__/../../tests/<recipe>; emulate by # is_enrolled reads through the single meta loader (rcust P1); point its tests/ root at the
# creating the layout under a fake harness dir and pointing __file__ there. # temp layout.
fake_harness = tmp_path / "runner" / "harness" monkeypatch.setattr(harness_meta, "TESTS_DIR", str(tmp_path / "tests"))
fake_harness.mkdir(parents=True)
monkeypatch.setattr(canonical, "__file__", str(fake_harness / "canonical.py"))
assert canonical.is_enrolled(recipe) is True assert canonical.is_enrolled(recipe) is True
(tests_dir / "recipe_meta.py").write_text("WARM_CANONICAL = False\n") (tests_dir / "recipe_meta.py").write_text("WARM_CANONICAL = False\n")
assert canonical.is_enrolled(recipe) is False assert canonical.is_enrolled(recipe) is False
@ -65,9 +64,7 @@ def test_registry_roundtrip(tmp_path, monkeypatch):
def test_enrolled_recipes_scans_meta(tmp_path, monkeypatch): def test_enrolled_recipes_scans_meta(tmp_path, monkeypatch):
# enrolled_recipes() lists recipes whose tests/<r>/recipe_meta.py sets WARM_CANONICAL=True. # enrolled_recipes() lists recipes whose tests/<r>/recipe_meta.py sets WARM_CANONICAL=True.
fake_harness = tmp_path / "runner" / "harness" monkeypatch.setattr(harness_meta, "TESTS_DIR", str(tmp_path / "tests"))
fake_harness.mkdir(parents=True)
monkeypatch.setattr(canonical, "__file__", str(fake_harness / "canonical.py"))
for name, body in ( for name, body in (
("aaa", "WARM_CANONICAL = True\n"), ("aaa", "WARM_CANONICAL = True\n"),
("bbb", "DEPS=['x']\n"), ("bbb", "DEPS=['x']\n"),

48
tests/unit/test_conftest_fixtures.py Normal file
View File

@ -0,0 +1,48 @@
"""Unit tests for the shared conftest fixtures added/reshaped by the rcust restructure (P2d/P4):
`op_state` (run-scoped op context from $CCCI_OP_STATE_FILE) and `deps` (consolidated dep creds
with attribute sugar). Pure — exercised via request.getfixturevalue with env monkeypatched."""
from __future__ import annotations
import json
import pytest
def test_op_state_fixture_reads_file(tmp_path, monkeypatch, request):
f = tmp_path / "op.json"
f.write_text(json.dumps({"backup": {"snapshot_id": "abc123"}, "upgrade": {"head_ref": "h"}}))
monkeypatch.setenv("CCCI_OP_STATE_FILE", str(f))
st = request.getfixturevalue("op_state")
assert st["backup"]["snapshot_id"] == "abc123"
assert st["upgrade"]["head_ref"] == "h"
def test_op_state_fixture_skips_without_env(monkeypatch, request):
monkeypatch.delenv("CCCI_OP_STATE_FILE", raising=False)
with pytest.raises(pytest.skip.Exception, match="orchestrator"):
request.getfixturevalue("op_state")
def test_op_state_fixture_skips_on_missing_file(tmp_path, monkeypatch, request):
monkeypatch.setenv("CCCI_OP_STATE_FILE", str(tmp_path / "nope.json"))
with pytest.raises(pytest.skip.Exception, match="missing"):
request.getfixturevalue("op_state")
def test_deps_fixture_entries_expose_attributes(tmp_path, monkeypatch, request):
"""`deps` (session-scoped) coerces the run deps file into entries with .domain/.realm/...
attribute sugar while keeping dict-style access (rcust P2d). Single test for the session-
cached fixture (one instantiation)."""
f = tmp_path / "deps.json"
f.write_text(
json.dumps(
{"keycloak": {"recipe": "keycloak", "domain": "kc.x", "client_secret": "s3cret"}}
)
)
monkeypatch.setenv("CCCI_DEPS_FILE", str(f))
deps = request.getfixturevalue("deps")
assert deps["keycloak"].domain == "kc.x"
assert deps["keycloak"]["client_secret"] == "s3cret"
with pytest.raises(AttributeError):
_ = deps["keycloak"].not_a_field

47
tests/unit/test_deps.py
View File

@ -1,9 +1,9 @@
"""Unit tests for runner/harness/deps.py (Phase 2 §4.2 / Q2.3). """Unit tests for runner/harness/deps.py (Phase 2 §4.2 / Q2.3).
Pure-Python: no real deploys. Tests the declarative parts of the dep resolver — declared_deps Pure-Python: no real deploys. Tests the declarative parts of the dep resolver — DEPS declaration
reading from `tests/<recipe>/recipe_meta.py`, the per-dep domain derivation, and write/load of the (read through the single meta loader since rcust P1), the per-dep domain derivation, and write/load
run state file. The deploy_deps + teardown_deps integration is exercised by real e2e against cc-ci of the run state file. The deploy_deps + teardown_deps integration is exercised by real e2e against
(Q2.4 acceptance). cc-ci (Q2.4 acceptance).
""" """
from __future__ import annotations from __future__ import annotations
@ -13,42 +13,23 @@ import sys
sys.path.insert(0, os.path.join(os.path.dirname(__file__), "..", "..", "runner")) sys.path.insert(0, os.path.join(os.path.dirname(__file__), "..", "..", "runner"))
from harness import deps # noqa: E402 from harness import deps # noqa: E402
from harness import meta as meta_mod # noqa: E402
def test_declared_deps_returns_empty_for_no_meta(monkeypatch, tmp_path): def test_declared_deps_empty_for_no_meta(monkeypatch, tmp_path):
"""A recipe with no recipe_meta.py returns [].""" """A recipe with no recipe_meta.py declares no deps (rcust P1: DEPS via meta.load)."""
fake_recipe = "ccci-no-meta" monkeypatch.setattr(meta_mod, "TESTS_DIR", str(tmp_path / "tests"))
# No file at tests/<fake_recipe>/recipe_meta.py -> declared_deps reads nothing -> [] assert meta_mod.load("ccci-no-meta").DEPS == []
monkeypatch.chdir(tmp_path)
assert deps.declared_deps(fake_recipe) == []
def test_declared_deps_reads_DEPS_list(tmp_path, monkeypatch): def test_declared_deps_reads_DEPS_list(tmp_path, monkeypatch):
"""A recipe_meta.py with `DEPS = [...]` returns the list.""" """A recipe_meta.py with `DEPS = [...]` surfaces the list on the loaded meta (the orchestrator
fake_recipe = "ccci-with-deps" reads meta.DEPS — the successor of the deleted deps.declared_deps loader)."""
# Build a fake repo layout under tmp_path recipe_dir = tmp_path / "tests" / "ccci-with-deps"
recipe_dir = tmp_path / "tests" / fake_recipe
recipe_dir.mkdir(parents=True) recipe_dir.mkdir(parents=True)
(recipe_dir / "recipe_meta.py").write_text('HEALTH_PATH = "/"\nDEPS = ["keycloak", "redis"]\n') (recipe_dir / "recipe_meta.py").write_text('HEALTH_PATH = "/"\nDEPS = ["keycloak", "redis"]\n')
# Patch the deps module's idea of "where the repo is" by monkey-patching __file__ for the monkeypatch.setattr(meta_mod, "TESTS_DIR", str(tmp_path / "tests"))
# function indirectly: declared_deps uses `os.path.dirname(__file__), "..", "..", "tests"` — assert meta_mod.load("ccci-with-deps").DEPS == ["keycloak", "redis"]
# which resolves to the real repo's `tests/`. So instead, override that with a symlink/dir
# under tmp_path: deps.__file__ points at the runner module. We can't easily relocate that.
# Instead, mock the path by writing the fake recipe under the REAL tests/ dir.
real_tests = os.path.join(os.path.dirname(deps.__file__), "..", "..", "tests")
target_dir = os.path.join(real_tests, fake_recipe)
os.makedirs(target_dir, exist_ok=True)
target_meta = os.path.join(target_dir, "recipe_meta.py")
try:
with open(target_meta, "w") as f:
f.write('DEPS = ["keycloak", "redis"]\n')
result = deps.declared_deps(fake_recipe)
assert result == ["keycloak", "redis"]
finally:
if os.path.exists(target_meta):
os.remove(target_meta)
if os.path.isdir(target_dir):
os.rmdir(target_dir)
def test_dep_domain_distinct_per_dep(): def test_dep_domain_distinct_per_dep():

13
tests/unit/test_discovery.py
View File

@ -71,17 +71,18 @@ def test_repo_local_wins_when_approved(tmp_path):
def test_custom_tests_repo_local_gated(tmp_path, monkeypatch): def test_custom_tests_repo_local_gated(tmp_path, monkeypatch):
# non-lifecycle test_*.py from repo-local only count for approved recipes; lifecycle names excluded # custom test_*.py from repo-local only count for approved recipes (HC2); placement rule
# (rcust P4): custom tests live under functional/ (or playwright/) — top-level files are
# lifecycle overlays only, so the repo-local custom here sits in functional/.
# Use a synthetic recipe name + monkeypatched cc_ci_dir so this is independent of what # Use a synthetic recipe name + monkeypatched cc_ci_dir so this is independent of what
# tests/<real-recipe>/ ships (Phase-2 custom-html now also ships functional/ + playwright/, # tests/<real-recipe>/ ships (F2-1).
# which would legitimately appear in custom_tests for "custom-html" — F2-1).
fake_recipe = "ccci-hc2-fixture" fake_recipe = "ccci-hc2-fixture"
monkeypatch.setattr(discovery, "cc_ci_dir", lambda r: str(tmp_path / "cc-ci" / r)) monkeypatch.setattr(discovery, "cc_ci_dir", lambda r: str(tmp_path / "cc-ci" / r))
(tmp_path / "cc-ci" / fake_recipe).mkdir(parents=True) (tmp_path / "cc-ci" / fake_recipe).mkdir(parents=True)
rl = tmp_path / "repo" rl = tmp_path / "repo"
rl.mkdir() (rl / "functional").mkdir(parents=True)
(rl / "test_sso.py").write_text("# repo-local custom\n") (rl / "functional" / "test_sso.py").write_text("# repo-local custom\n")
(rl / "test_install.py").write_text("# lifecycle name -> excluded from custom\n") (rl / "functional" / "test_install.py").write_text("# lifecycle name -> excluded from custom\n")
_approve(tmp_path) # not approved -> repo-local custom ignored _approve(tmp_path) # not approved -> repo-local custom ignored
assert discovery.custom_tests(fake_recipe, str(rl)) == [] assert discovery.custom_tests(fake_recipe, str(rl)) == []

16
tests/unit/test_discovery_phase2.py
View File

@ -1,6 +1,6 @@
"""Unit tests for Phase-2 discovery additions (plan §4.1). """Unit tests for Phase-2 discovery additions (plan §4.1).
Proves the `custom_tests` discovery recurses into the per-recipe `functional/` + `playwright/` Proves the `custom_tests` discovery covers exactly the per-recipe `functional/` + `playwright/`
subdirs as well as the top-level dir, while still excluding lifecycle `test_<op>.py` names and subdirs as well as the top-level dir, while still excluding lifecycle `test_<op>.py` names and
honouring the HC2 repo-local approval gate. honouring the HC2 repo-local approval gate.
@ -27,16 +27,16 @@ def teardown_function():
os.environ.pop("CCCI_REPO_LOCAL_APPROVED_FILE", None) os.environ.pop("CCCI_REPO_LOCAL_APPROVED_FILE", None)
def test_custom_tests_recurses_functional_and_playwright(tmp_path, monkeypatch): def test_custom_tests_placement_rule_functional_playwright_only(tmp_path, monkeypatch):
"""A Phase-2 cc-ci recipe layout: functional/test_*.py + playwright/test_*.py + top-level """Placement rule (rcust P4): custom tests are discovered ONLY under functional/ +
test_*.py — all are discovered as custom tests; the lifecycle names are excluded.""" playwright/. A top-level non-lifecycle test_*.py is NOT discovered (top level is reserved
for lifecycle overlays); lifecycle names inside the subdirs stay excluded (defensive)."""
# Point cc-ci's per-recipe dir at a fake recipe in tmp_path # Point cc-ci's per-recipe dir at a fake recipe in tmp_path
fake_recipe = "ccci-phase2-fixture" fake_recipe = "ccci-phase2-fixture"
fake_dir = tmp_path / "tests" / fake_recipe fake_dir = tmp_path / "tests" / fake_recipe
(fake_dir / "functional").mkdir(parents=True) (fake_dir / "functional").mkdir(parents=True)
(fake_dir / "playwright").mkdir() (fake_dir / "playwright").mkdir()
# legitimate custom tests at multiple levels (fake_dir / "test_sso_smoke.py").write_text("# top-level — NOT discovered since P4\n")
(fake_dir / "test_sso_smoke.py").write_text("# top-level cross-cutting\n")
(fake_dir / "functional" / "test_health_check.py").write_text("# parity port\n") (fake_dir / "functional" / "test_health_check.py").write_text("# parity port\n")
(fake_dir / "functional" / "test_content_roundtrip.py").write_text("# recipe-specific\n") (fake_dir / "functional" / "test_content_roundtrip.py").write_text("# recipe-specific\n")
(fake_dir / "playwright" / "test_login_flow.py").write_text("# UI flow\n") (fake_dir / "playwright" / "test_login_flow.py").write_text("# UI flow\n")
@ -49,11 +49,11 @@ def test_custom_tests_recurses_functional_and_playwright(tmp_path, monkeypatch):
customs = discovery.custom_tests(fake_recipe, None) customs = discovery.custom_tests(fake_recipe, None)
names = sorted((src, os.path.basename(p)) for src, p in customs) names = sorted((src, os.path.basename(p)) for src, p in customs)
# Top-level + functional/ + playwright/ all discovered; lifecycle name excluded # functional/ + playwright/ discovered; top-level custom + lifecycle name are NOT
assert ("cc-ci", "test_sso_smoke.py") in names
assert ("cc-ci", "test_health_check.py") in names assert ("cc-ci", "test_health_check.py") in names
assert ("cc-ci", "test_content_roundtrip.py") in names assert ("cc-ci", "test_content_roundtrip.py") in names
assert ("cc-ci", "test_login_flow.py") in names assert ("cc-ci", "test_login_flow.py") in names
assert ("cc-ci", "test_sso_smoke.py") not in names
assert ("cc-ci", "test_install.py") not in names assert ("cc-ci", "test_install.py") not in names

2
tests/unit/test_f211_sso_skip.py
View File

@ -30,7 +30,7 @@ def test_sso_dep_unverified_true_when_declared_notready_and_skipped():
def test_sso_dep_unverified_false_when_deps_ready(): def test_sso_dep_unverified_false_when_deps_ready():
"""deps ready (setup_custom_tests succeeded) → SSO tests actually ran → not a failure.""" """deps ready (dep provisioning succeeded) → SSO tests actually ran → not a failure."""
assert not run_recipe_ci.sso_dep_unverified( assert not run_recipe_ci.sso_dep_unverified(
["keycloak"], deps_ready=True, requires_deps_skipped=0 ["keycloak"], deps_ready=True, requires_deps_skipped=0
) )

18
tests/unit/test_f212_upgrade_convergence.py
View File

@ -14,6 +14,7 @@ So `-c` + owned-wait is non-vacuous: a genuinely-broken upgrade stays RED.
from __future__ import annotations from __future__ import annotations
import dataclasses
import os import os
import sys import sys
@ -21,6 +22,7 @@ import pytest
sys.path.insert(0, os.path.join(os.path.dirname(__file__), "..", "..", "runner")) sys.path.insert(0, os.path.join(os.path.dirname(__file__), "..", "..", "runner"))
from harness import lifecycle as lc # noqa: E402 from harness import lifecycle as lc # noqa: E402
from harness import meta as harness_meta # noqa: E402
def _fake_clock(monkeypatch): def _fake_clock(monkeypatch):
@ -31,11 +33,15 @@ def _fake_clock(monkeypatch):
return state return state
_DRIVE_META = { # RecipeMeta (rcust P1: wait_ready_probes reads meta.READY_PROBE off the loaded object); defaults
"READY_PROBE": lambda d: [ # + the drive-style probe hook (P3 ctx signature: the probe receives a HookCtx).
{"host": f"collabora-{d}", "path": "/hosting/discovery", "ok": (200,)} _DRIVE_META = dataclasses.replace(
] harness_meta.load("ccci-no-such-recipe"),
} READY_PROBE=lambda ctx: [
{"host": f"collabora-{ctx.domain}", "path": "/hosting/discovery", "ok": (200,)}
],
)
_NO_PROBE_META = harness_meta.load("ccci-no-such-recipe")
def test_wait_ready_probes_raises_when_never_ready(monkeypatch): def test_wait_ready_probes_raises_when_never_ready(monkeypatch):
@ -57,7 +63,7 @@ def test_wait_ready_probes_returns_when_ready(monkeypatch):
def test_wait_ready_probes_noop_without_probe(monkeypatch): def test_wait_ready_probes_noop_without_probe(monkeypatch):
"""A recipe with no READY_PROBE is a clean no-op (default behavior preserved for all recipes).""" """A recipe with no READY_PROBE is a clean no-op (default behavior preserved for all recipes)."""
monkeypatch.setattr(lc, "http_get", lambda *a, **k: 599) # would fail if it were consulted monkeypatch.setattr(lc, "http_get", lambda *a, **k: 599) # would fail if it were consulted
lc.wait_ready_probes({}, "x.ci.commoninternet.net", timeout=1) # no raise, no call lc.wait_ready_probes(_NO_PROBE_META, "x.ci.commoninternet.net", timeout=1) # no raise, no call
def test_wait_healthy_raises_when_services_never_converge(monkeypatch): def test_wait_healthy_raises_when_services_never_converge(monkeypatch):

177
tests/unit/test_manifest.py Normal file
View File

@ -0,0 +1,177 @@
"""Unit tests for the customization manifest (rcust P5; spec §8 R4 mitigation).
The manifest is PURE PRESENTATION (must never influence a verdict); these tests pin that it is
COMPLETE (every customization surface a synthetic recipe exercises shows up), DETERMINISTIC
(same inputs -> byte-identical JSON), serializable, and HC2-honoring (unapproved repo-local
contributions are invisible). Pure / tmp-file only. Run cold:
cc-ci-run -m pytest tests/unit/test_manifest.py -q
"""
from __future__ import annotations
import json
import os
import sys
sys.path.insert(0, os.path.join(os.path.dirname(__file__), "..", "..", "runner"))
from harness import discovery, manifest # noqa: E402
from harness import meta as meta_mod # noqa: E402
RECIPE = "ccci-manifest-fixture"
def _mk_synthetic(tmp_path, monkeypatch, approved=True):
"""A synthetic recipe dir exercising EVERY manifest surface, plus a repo-local tests dir.
cc-ci side: meta (2 data keys + 1 hook key non-default), ops.py (2 pre-ops), install_steps.sh,
compose.ccci.yml, test_backup.py overlay, 2 functional + 1 playwright custom tests.
repo-local side: test_restore.py overlay + 1 functional custom test (visible iff approved, HC2).
"""
ccci_root = tmp_path / "cc-ci-tests"
d = ccci_root / RECIPE
(d / "functional").mkdir(parents=True)
(d / "playwright").mkdir()
(d / "recipe_meta.py").write_text(
"HTTP_TIMEOUT = 600\n"
"DEPS = ['keycloak']\n"
"def EXTRA_ENV(ctx):\n return {}\n"
"_PRIVATE = 'exempt'\n"
)
(d / "ops.py").write_text("def pre_upgrade(ctx):\n pass\n\ndef pre_backup(ctx):\n pass\n")
(d / "install_steps.sh").write_text("#!/usr/bin/env bash\n")
(d / "compose.ccci.yml").write_text("version: '3.8'\n")
(d / "test_backup.py").write_text("# lifecycle overlay\n")
(d / "functional" / "test_a.py").write_text("# custom\n")
(d / "functional" / "test_b.py").write_text("# custom\n")
(d / "playwright" / "test_ui.py").write_text("# custom\n")
rl = tmp_path / "repo-local"
(rl / "functional").mkdir(parents=True)
(rl / "functional" / "test_c.py").write_text("# repo-local custom\n")
(rl / "test_restore.py").write_text("# repo-local lifecycle overlay\n")
monkeypatch.setattr(discovery, "cc_ci_dir", lambda r: str(ccci_root / r))
monkeypatch.setattr(meta_mod, "TESTS_DIR", str(ccci_root)) # compose.ccci.yml discovery
approved_file = tmp_path / "approved.txt"
approved_file.write_text(f"{RECIPE}\n" if approved else "")
monkeypatch.setenv("CCCI_REPO_LOCAL_APPROVED_FILE", str(approved_file))
meta = meta_mod.load(RECIPE, tests_dir=str(ccci_root))
return meta, str(rl)
def test_manifest_complete(tmp_path, monkeypatch):
# Every surface the synthetic recipe customizes appears — nothing silently dropped (R4).
meta, rl = _mk_synthetic(tmp_path, monkeypatch)
m = manifest.build(RECIPE, meta, rl)
assert m["meta_non_default"] == {
"DEPS": ["keycloak"],
"EXTRA_ENV": "<hook>",
"HTTP_TIMEOUT": 600,
}
assert m["hooks"] == {
"ops.py": {"cc-ci": ["pre_backup", "pre_upgrade"]},
"install_steps.sh": "cc-ci",
"compose.ccci.yml": "cc-ci",
}
assert m["overlays"] == {"backup": "cc-ci", "restore": "repo-local"}
assert m["custom_tests"] == {
"cc-ci": {"functional": 2, "playwright": 1},
"repo-local": {"functional": 1},
}
assert m["env_overrides"] == []
def test_manifest_deterministic_and_serializable(tmp_path, monkeypatch):
meta, rl = _mk_synthetic(tmp_path, monkeypatch)
a = manifest.build(RECIPE, meta, rl)
b = manifest.build(RECIPE, meta, rl)
assert json.dumps(a, sort_keys=True) == json.dumps(b, sort_keys=True)
assert json.loads(json.dumps(a)) == a # round-trips: no callables/tuples leak through
def test_manifest_zero_config_floor(tmp_path, monkeypatch):
# A recipe with NO customization at all -> every section empty, render says so explicitly.
ccci_root = tmp_path / "cc-ci-tests"
(ccci_root / RECIPE).mkdir(parents=True)
monkeypatch.setattr(discovery, "cc_ci_dir", lambda r: str(ccci_root / r))
monkeypatch.setattr(meta_mod, "TESTS_DIR", str(ccci_root))
monkeypatch.setenv("CCCI_REPO_LOCAL_APPROVED_FILE", str(tmp_path / "missing.txt"))
meta = meta_mod.load(RECIPE, tests_dir=str(ccci_root))
m = manifest.build(RECIPE, meta, None)
assert m == {
"meta_non_default": {},
"hooks": {},
"overlays": {},
"custom_tests": {},
"env_overrides": [],
}
out = manifest.render(RECIPE, m)
assert f"===== customization manifest: {RECIPE} =====" in out
assert "(none — zero-config floor)" in out
def test_manifest_repo_local_hc2_gate(tmp_path, monkeypatch):
# Unapproved recipe -> repo-local overlay + custom tests INVISIBLE (same default-deny as the
# discovery they ride on; the manifest must not advertise code the run will not execute).
meta, rl = _mk_synthetic(tmp_path, monkeypatch, approved=False)
m = manifest.build(RECIPE, meta, rl)
assert m["overlays"] == {"backup": "cc-ci"} # repo-local test_restore.py gone
assert "repo-local" not in m["custom_tests"]
def test_manifest_env_overrides_and_ci_flag(tmp_path, monkeypatch):
meta, rl = _mk_synthetic(tmp_path, monkeypatch)
monkeypatch.setenv("CCCI_SKIP_GENERIC_BACKUP", "1")
monkeypatch.setenv("CCCI_SKIP_GENERIC_UPGRADE", "0") # falsy -> not an active override
m = manifest.build(RECIPE, meta, rl)
assert m["env_overrides"] == ["CCCI_SKIP_GENERIC_BACKUP"]
monkeypatch.delenv("DRONE", raising=False)
assert "!!" not in manifest.render(RECIPE, m) # local dev: no CI warning
monkeypatch.setenv("DRONE", "true") # riding a CI run -> loud flag (P2c)
assert "!! dev-only override active in CI" in manifest.render(RECIPE, m)
def test_manifest_redacts_sensitive_named_values(tmp_path, monkeypatch):
# Meta values are repo-public by construction, but the manifest lands on the dashboard:
# secret-NAMED entries (top-level or nested dict keys, e.g. plausible's
# EXTRA_ENV["SECRET_KEY_BASE"] dummy) render as '<redacted>' — name shown, value masked.
# Non-sensitive names (incl. KEYCLOAK_* — 'KEY' matches only as a word segment) pass through.
ccci_root = tmp_path / "cc-ci-tests"
d = ccci_root / RECIPE
d.mkdir(parents=True)
(d / "recipe_meta.py").write_text(
"EXTRA_ENV = {\n"
" 'SECRET_KEY_BASE': 'dummy-ci-constant',\n"
" 'API_KEY': 'also-dummy',\n"
" 'KEYCLOAK_URL': 'https://kc.example',\n"
"}\n"
)
monkeypatch.setattr(discovery, "cc_ci_dir", lambda r: str(ccci_root / r))
monkeypatch.setattr(meta_mod, "TESTS_DIR", str(ccci_root))
monkeypatch.setenv("CCCI_REPO_LOCAL_APPROVED_FILE", str(tmp_path / "missing.txt"))
meta = meta_mod.load(RECIPE, tests_dir=str(ccci_root))
m = manifest.build(RECIPE, meta, None)
assert m["meta_non_default"]["EXTRA_ENV"] == {
"SECRET_KEY_BASE": "<redacted>",
"API_KEY": "<redacted>",
"KEYCLOAK_URL": "https://kc.example",
}
out = manifest.render(RECIPE, m)
assert "dummy-ci-constant" not in out and "also-dummy" not in out
assert "SECRET_KEY_BASE" in out # the key NAME stays visible
def test_render_lists_every_surface(tmp_path, monkeypatch):
meta, rl = _mk_synthetic(tmp_path, monkeypatch)
out = manifest.render(RECIPE, manifest.build(RECIPE, meta, rl))
lines = out.splitlines()
assert lines[0] == f"===== customization manifest: {RECIPE} ====="
assert "meta (non-default): DEPS=['keycloak'] EXTRA_ENV='<hook>' HTTP_TIMEOUT=600" in lines
assert (
"hooks: ops.py[pre_backup,pre_upgrade](cc-ci) install_steps.sh(cc-ci) compose.ccci.yml(cc-ci)"
in lines
)
assert "overlays: test_backup.py(cc-ci) test_restore.py(repo-local)" in lines
assert "custom tests: functional/=2 playwright/=1 (cc-ci) functional/=1 (repo-local)" in lines
assert "env overrides: (none)" in lines

276
tests/unit/test_meta.py Normal file
View File

@ -0,0 +1,276 @@
"""Unit tests for the single recipe-meta loader + key registry (rcust P1; spec §8 R1/R6).
Covers: every in-repo recipe_meta.py loads clean through the registry (THE typo gate), validation
hard-errors (unknown key, wrong type, callable on a data key), the zero-config baseline defaults
(spec §2), the underscore exemption for recipe-private constants, and the registry↔generated-doc
sync (P1.5; drift fails CI). Run: cc-ci-run -m pytest tests/unit/test_meta.py -q
"""
from __future__ import annotations
import os
import subprocess
import sys
import pytest
sys.path.insert(0, os.path.join(os.path.dirname(__file__), "..", "..", "runner"))
from harness import meta as meta_mod # noqa: E402
from harness.meta import KEYS, MetaError, RecipeMeta # noqa: E402
ROOT = os.path.dirname(os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
def _recipes_with_meta() -> list[str]:
tests_dir = os.path.join(ROOT, "tests")
return sorted(
n
for n in os.listdir(tests_dir)
if os.path.isfile(os.path.join(tests_dir, n, "recipe_meta.py"))
)
# ---- the typo gate: every in-repo recipe meta must validate against the registry --------------
@pytest.mark.parametrize("recipe", _recipes_with_meta())
def test_every_recipe_meta_loads_clean(recipe):
"""All tests/*/recipe_meta.py in the repo load + validate through the registry. A typo'd or
unregistered ALL-CAPS key in any recipe meta fails HERE, at PR time — not silently at run
time (the R6 failure mode this restructure kills)."""
meta = meta_mod.load(recipe)
assert isinstance(meta, RecipeMeta)
# sanity: the 4 base keys always materialize with usable types
assert isinstance(meta.HEALTH_PATH, str)
assert isinstance(meta.HEALTH_OK, tuple) and meta.HEALTH_OK
assert isinstance(meta.DEPLOY_TIMEOUT, int) and isinstance(meta.HTTP_TIMEOUT, int)
# ---- zero-config baseline (spec §2) ------------------------------------------------------------
def test_missing_meta_yields_spec_baseline(tmp_path):
meta = meta_mod.load("no-such-recipe", tests_dir=str(tmp_path))
assert meta.HEALTH_PATH == "/"
assert meta.HEALTH_OK == (200, 301, 302)
assert meta.DEPLOY_TIMEOUT == 600
assert meta.HTTP_TIMEOUT == 300
assert meta.BACKUP_CAPABLE is None # None = auto-detect (tri-state, not False)
assert meta.EXPECTED_NA is None
assert meta.READY_PROBE is None
assert meta.UPGRADE_BASE_VERSION is None
assert meta.BACKUP_VERIFY is None
assert meta.UPGRADE_EXTRA_ENV is None
assert meta.EXTRA_ENV == {}
assert meta.DEPS == []
assert meta.WARM_CANONICAL is False
assert meta.SCREENSHOT is None
assert meta_mod.non_default(meta) == {}
def test_registry_field_set_matches_dataclass():
"""The RecipeMeta field set is generated from KEYS — no drift possible, pinned anyway."""
import dataclasses
assert [f.name for f in dataclasses.fields(RecipeMeta)] == [k.name for k in KEYS]
# the 14 final keys, no more (the 3 P2-deleted legacy keys are gone from the registry,
# so any recipe_meta still setting them hard-fails the typo gate)
assert len(KEYS) == 14
assert not [k for k in KEYS if k.deprecated]
for gone in ("CHAOS_BASE_DEPLOY", "OIDC_AT_INSTALL", "SKIP_GENERIC"):
assert gone not in {k.name for k in KEYS}
# ---- validation hard errors (locked decision: fail fast at load) -------------------------------
def _write_meta(tmp_path, body: str, recipe: str = "r") -> str:
d = tmp_path / recipe
d.mkdir(exist_ok=True)
(d / "recipe_meta.py").write_text(body)
return recipe
def test_unknown_key_raises_with_suggestion(tmp_path):
r = _write_meta(tmp_path, "READINESS_PROBE = None\n") # the R6 typo example
with pytest.raises(MetaError) as ei:
meta_mod.load(r, tests_dir=str(tmp_path))
msg = str(ei.value)
assert "READINESS_PROBE" in msg and "READY_PROBE" in msg # names the typo + nearest key
def test_unknown_key_without_near_match_lists_registry(tmp_path):
r = _write_meta(tmp_path, "TOTALLY_BOGUS_KNOB = 1\n")
with pytest.raises(MetaError) as ei:
meta_mod.load(r, tests_dir=str(tmp_path))
assert "HEALTH_PATH" in str(ei.value) # registered keys listed for the reader
def test_wrong_type_raises(tmp_path):
r = _write_meta(tmp_path, 'DEPLOY_TIMEOUT = "900"\n')
with pytest.raises(MetaError, match="DEPLOY_TIMEOUT"):
meta_mod.load(r, tests_dir=str(tmp_path))
def test_bool_not_accepted_as_int(tmp_path):
r = _write_meta(tmp_path, "DEPLOY_TIMEOUT = True\n")
with pytest.raises(MetaError, match="DEPLOY_TIMEOUT"):
meta_mod.load(r, tests_dir=str(tmp_path))
def test_callable_on_data_key_rejected(tmp_path):
r = _write_meta(tmp_path, "def HEALTH_PATH():\n return '/'\n")
with pytest.raises(MetaError, match="hook-typed"):
meta_mod.load(r, tests_dir=str(tmp_path))
def test_non_callable_on_hook_key_rejected(tmp_path):
r = _write_meta(tmp_path, "READY_PROBE = ['not', 'a', 'callable']\n")
with pytest.raises(MetaError, match="READY_PROBE"):
meta_mod.load(r, tests_dir=str(tmp_path))
def test_underscore_names_are_private_and_exempt(tmp_path):
r = _write_meta(
tmp_path,
"_WELCOME_TEXT_MARKER = 'marker-xyz'\n_MAX_USERS = 42\n"
"EXTRA_ENV = {'WELCOME_TEXT': _WELCOME_TEXT_MARKER, 'USERS': str(_MAX_USERS)}\n",
)
meta = meta_mod.load(r, tests_dir=str(tmp_path))
assert meta.EXTRA_ENV == {"WELCOME_TEXT": "marker-xyz", "USERS": "42"}
def test_lowercase_helpers_ignored(tmp_path):
r = _write_meta(
tmp_path,
"def _helper(d):\n return {'K': d}\n\ndef EXTRA_ENV(ctx):\n return _helper(ctx.domain)\n",
)
meta = meta_mod.load(r, tests_dir=str(tmp_path))
ctx = meta_mod.hook_ctx("x.example", meta)
assert meta_mod.extra_env(meta, ctx) == {"K": "x.example"}
# ---- normalization + helpers --------------------------------------------------------------------
def test_health_ok_list_normalized_to_tuple(tmp_path):
r = _write_meta(tmp_path, "HEALTH_OK = [200, 302]\n")
assert meta_mod.load(r, tests_dir=str(tmp_path)).HEALTH_OK == (200, 302)
def test_extra_env_dict_and_callable_forms(tmp_path):
r = _write_meta(tmp_path, "EXTRA_ENV = {'A': 1}\n")
meta = meta_mod.load(r, tests_dir=str(tmp_path))
assert meta_mod.extra_env(meta, meta_mod.hook_ctx("d", meta)) == {"A": "1"} # stringified
r2 = _write_meta(
tmp_path, "UPGRADE_EXTRA_ENV = lambda ctx: {'COMPOSE_FILE': ctx.domain}\n", recipe="r2"
)
meta2 = meta_mod.load(r2, tests_dir=str(tmp_path))
ctx2 = meta_mod.hook_ctx("dom.x", meta2, op="upgrade")
assert meta_mod.upgrade_extra_env(meta2, ctx2) == {"COMPOSE_FILE": "dom.x"}
assert meta_mod.extra_env(meta2, ctx2) == {} # unset EXTRA_ENV resolves to {}
# ---- P3: uniform ctx hook convention -------------------------------------------------------------
def test_hook_ctx_fields(tmp_path):
meta = meta_mod.load("no-such", tests_dir=str(tmp_path))
ctx = meta_mod.hook_ctx("app.ci.example", meta, op="backup")
assert ctx.domain == "app.ci.example"
assert ctx.base_url == "https://app.ci.example"
assert ctx.meta is meta
assert ctx.op == "backup"
assert meta_mod.hook_ctx("d", meta).op is None
def test_hook_ctx_deps_from_run_file(tmp_path, monkeypatch):
import json
meta = meta_mod.load("no-such", tests_dir=str(tmp_path))
monkeypatch.delenv("CCCI_DEPS_FILE", raising=False)
assert meta_mod.hook_ctx("d", meta).deps is None
f = tmp_path / "deps.json"
f.write_text(json.dumps({"keycloak": {"recipe": "keycloak", "domain": "kc.x"}}))
monkeypatch.setenv("CCCI_DEPS_FILE", str(f))
deps = meta_mod.hook_ctx("d", meta).deps
assert deps["keycloak"]["domain"] == "kc.x"
f.write_text("{}") # empty dict -> None (deps declared but not provisioned)
assert meta_mod.hook_ctx("d", meta).deps is None
def test_legacy_hook_signature_raises_clear_meta_error(tmp_path):
"""A pre-restructure hook signature must fail AT LOAD with a migration message — never a
silent TypeError mid-run (P3.4)."""
r = _write_meta(tmp_path, "def READY_PROBE(domain):\n return []\n")
with pytest.raises(MetaError, match="ctx"):
meta_mod.load(r, tests_dir=str(tmp_path))
r2 = _write_meta(tmp_path, "EXTRA_ENV = lambda domain: {}\n", recipe="r2")
with pytest.raises(MetaError, match="restructure"):
meta_mod.load(r2, tests_dir=str(tmp_path))
r3 = _write_meta(
tmp_path, "def SCREENSHOT(page, domain, meta):\n return None\n", recipe="r3"
)
with pytest.raises(MetaError, match="page, ctx"):
meta_mod.load(r3, tests_dir=str(tmp_path))
def test_ctx_hook_signatures_accepted(tmp_path):
r = _write_meta(
tmp_path,
"def READY_PROBE(ctx):\n return []\n"
"def BACKUP_VERIFY(ctx):\n return True\n"
"def SCREENSHOT(page, ctx):\n return None\n"
"def EXTRA_ENV(ctx):\n return {}\n",
)
meta = meta_mod.load(r, tests_dir=str(tmp_path))
assert callable(meta.READY_PROBE) and callable(meta.SCREENSHOT)
def test_check_hook_signature_for_pre_op_hooks():
"""The orchestrator validates ops.py pre_<op> hooks with the same checker (legacy
(domain, meta) form names the migration)."""
def legacy(domain, meta):
pass
def new(ctx):
pass
with pytest.raises(MetaError, match="ctx"):
meta_mod.check_hook_signature(legacy, ("ctx",), "tests/x/ops.py::pre_upgrade")
meta_mod.check_hook_signature(new, ("ctx",), "tests/x/ops.py::pre_upgrade") # no raise
def test_non_default_reports_only_customized_keys(tmp_path):
r = _write_meta(tmp_path, "DEPLOY_TIMEOUT = 1500\nDEPS = ['keycloak']\n")
nd = meta_mod.non_default(meta_mod.load(r, tests_dir=str(tmp_path)))
assert nd == {"DEPLOY_TIMEOUT": 1500, "DEPS": ["keycloak"]}
def test_meta_is_frozen():
import dataclasses
meta = meta_mod.load("custom-html")
with pytest.raises(dataclasses.FrozenInstanceError):
meta.DEPLOY_TIMEOUT = 1
# ---- doc generation sync (P1.5: the committed §4 table == the registry rendering) ---------------
def test_generated_doc_table_in_sync():
"""docs/recipe-customization.md's key reference table is GENERATED from the registry
(scripts/gen-meta-docs.py). If this fails: re-run `python3 scripts/gen-meta-docs.py` and
commit the result — the table must never drift from the registry (R5)."""
gen = os.path.join(ROOT, "scripts", "gen-meta-docs.py")
doc = os.path.join(ROOT, "docs", "recipe-customization.md")
rendered = subprocess.run(
[sys.executable, gen, "--print"], capture_output=True, text=True, check=True
).stdout
with open(doc) as f:
committed = f.read()
assert rendered.strip() in committed, (
"docs/recipe-customization.md key table is out of sync with the harness.meta registry — "
"run `python3 scripts/gen-meta-docs.py` and commit"
)

35
tests/unit/test_results.py
View File

@ -280,6 +280,41 @@ def test_build_results_threads_expected_na(tmp_path):
) # backup_restore declared; functional passed → clean ) # backup_restore declared; functional passed → clean
def test_build_results_threads_customization(tmp_path):
# rcust P5: the run-start customization manifest lands verbatim under "customization";
# omitted -> explicit None (key always present in the schema).
recs = [
{
"tier": "install",
"source": "generic",
"file": "g/test_install.py",
"rc": 0,
"junit": _write(tmp_path, "i.xml", JUNIT_PASS),
},
]
cust = {
"meta_non_default": {"HTTP_TIMEOUT": 600},
"hooks": {"install_steps.sh": "cc-ci"},
"overlays": {},
"custom_tests": {"cc-ci": {"functional": 2}},
"env_overrides": [],
}
kwargs = {
"recipe": "hedgedoc",
"version": "1.2.3",
"pr": "7",
"ref": None,
"records": recs,
"results": _results(),
"backup_capable": True,
"clean_teardown": True,
"no_secret_leak": True,
"finished_ts": 0.0,
}
assert R.build_results(**kwargs, customization=cust)["customization"] == cust
assert R.build_results(**kwargs)["customization"] is None
def test_write_results_roundtrip(tmp_path): def test_write_results_roundtrip(tmp_path):
data = {"run_id": "42", "level": 3, "stages": []} data = {"run_id": "42", "level": 3, "stages": []}
path = R.write_results(data, runs_dir_override=str(tmp_path)) path = R.write_results(data, runs_dir_override=str(tmp_path))

17
tests/unit/test_screenshot.py
View File

@ -11,6 +11,7 @@ import os
import sys import sys
sys.path.insert(0, os.path.join(os.path.dirname(__file__), "..", "..", "runner")) sys.path.insert(0, os.path.join(os.path.dirname(__file__), "..", "..", "runner"))
from harness import meta as meta_mod # noqa: E402
from harness import screenshot as S # noqa: E402 from harness import screenshot as S # noqa: E402
@ -29,3 +30,19 @@ def test_hook_returned_when_callable():
pass pass
assert S._load_screenshot_hook({"SCREENSHOT": hook}) is hook assert S._load_screenshot_hook({"SCREENSHOT": hook}) is hook
def test_screenshot_reachable_through_real_load_path(tmp_path):
"""R2 proof (rcust P1): a recipe SCREENSHOT hook declared in recipe_meta.py arrives at
screenshot._load_screenshot_hook through the REAL orchestrator load path (meta.load — the
object run_recipe_ci passes to capture()). Under the old six-loader world the orchestrator's
L1 allowlist dropped SCREENSHOT, so the hook was unreachable (spec §8 R2)."""
d = tmp_path / "shotrecipe"
d.mkdir()
(d / "recipe_meta.py").write_text(
"def SCREENSHOT(page, ctx):\n return None\n",
)
meta = meta_mod.load("shotrecipe", tests_dir=str(tmp_path))
hook = S._load_screenshot_hook(meta)
assert callable(hook), "SCREENSHOT hook did not survive the orchestrator load path (R2)"
assert S._load_screenshot_hook(meta_mod.load("no-such", tests_dir=str(tmp_path))) is None