From b2151af53223f79d489c8437b0e5a9b3694fa058 Mon Sep 17 00:00:00 2001 From: autonomic-bot Date: Thu, 28 May 2026 09:50:13 +0100 Subject: [PATCH] =?UTF-8?q?docs(2):=20Q5.1=20partial=20=E2=80=94=20enroll-?= =?UTF-8?q?recipe.md=20Phase-2=20contract?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Adds: - §2 layout: PARITY.md / functional/ / playwright/ subdirs (Phase 2 §4.1) - §2.1 Phase-2 contract: parity port + ≥2 specific functional tests + Playwright; custom-tier discovery from functional/ + playwright/; SOURCE comment audit - §2.2 DEPS = [...] declaration; orchestrator dep deploy order; deps_apps fixture; expected deploy-count = 1 + len(DEPS); F2-5 verify=True teardown - §2.3 harness.sso primitives (setup_keycloak_realm, oidc_password_grant, assert_discovery_endpoint); F2-7 note that setup is keycloak-specific - Worked example: lasuite-docs full Phase-2 layout (DEPS + functional/ + lifecycle overlays) and the !testme flow walked through end-to-end - Updated 'Run locally' to include restore + custom stages A new engineer can add a recipe's full Phase-2 suite from the docs alone (P8). Co-Authored-By: Claude Opus 4.7 (1M context) --- docs/enroll-recipe.md | 117 +++++++++++++++++++++++++++++++++++++++++- 1 file changed, 115 insertions(+), 2 deletions(-) diff --git a/docs/enroll-recipe.md b/docs/enroll-recipe.md index 18a03f9..94ae064 100644 --- a/docs/enroll-recipe.md +++ b/docs/enroll-recipe.md @@ -19,7 +19,14 @@ tests// ├── test_install.py # optional install overlay (runs ADDITIVELY alongside generic) ├── test_upgrade.py # optional upgrade overlay (runs ADDITIVELY alongside generic) ├── test_backup.py # optional backup overlay (runs ADDITIVELY alongside generic) -└── test_restore.py # optional restore overlay (runs ADDITIVELY alongside generic) +├── test_restore.py # optional restore overlay (runs ADDITIVELY alongside generic) +├── PARITY.md # Phase 2 P2: mapping table (recipe-maintainer tests → cc-ci tests) +├── functional/ # Phase 2 P3: parity ports + ≥2 NEW recipe-specific tests +│ ├── test_health_check.py # parity port of recipe-info//tests/health_check.py +│ ├── test_.py # ≥2 NEW recipe-specific functional tests +│ └── … +└── playwright/ # Phase 2 P6: browser flows where the app's core UX is a UI + └── test_.py ``` **A recipe is testable with ZERO config:** with no overlay files, the **generic lifecycle suite** @@ -54,6 +61,84 @@ Useful `harness.lifecycle` helpers for overlays: `http_get`, `http_fetch`, `http ops themselves are orchestrator-owned (you never call them from an overlay). The harness forces `LETS_ENCRYPT_ENV=""` (no ACME), a unique short domain per run, and guarantees teardown. +### 2.1 Phase-2 contract: parity port + recipe-specific functional tests + Playwright + +Beyond the lifecycle overlays, each recipe carries (plan §4.1): + +- **`PARITY.md`** — a mapping table from every `references/recipe-maintainer/recipe-info// + tests/*.py` to a comparable cc-ci test under `tests//functional/`, asserting the + *same thing* (not a renamed file). A deliberate non-port is documented in `DECISIONS.md` with + a technical reason — never a silent omission. +- **`functional/`** — parity-port tests + **≥2 NEW recipe-specific functional tests** that + exercise the app's characteristic behavior (per plan §4.3 — e.g. "create-an-object + + read-it-back, and one more that touches a distinctive feature"). Each parity-port file carries + a `SOURCE = "recipe-info//tests/"` comment near the top so audit is in-file. +- **`playwright/`** — browser flows where the recipe's core UX is a UI (P6). + +The orchestrator's **custom** tier discovers `test_*.py` in `tests//{functional,playwright}/` +(recursive, via `runner/harness/discovery.custom_tests`) and runs each as its own pytest against +the same `live_app` shared deployment. Lifecycle-named files (`test_install.py`/etc.) are +**excluded** from the custom tier — they live at the top level and run as lifecycle overlays. + +### 2.2 Recipe-test dependencies — DEPS = [...] (Phase 2 Q2.3) + +If your recipe needs other recipes deployed alongside it (an SSO provider, a database), declare +them in `recipe_meta.py`: + +```python +DEPS = ["keycloak"] # one entry per dep recipe name (cc-ci tests// must exist + work) +``` + +The orchestrator (plan §4.2): +1. Reads `DEPS` BEFORE deploying the recipe under test. +2. Deploys each dep at a per-run domain `-<6hex>.ci.commoninternet.net` (the 6hex is + hashed from `parent_recipe + pr + ref + dep_recipe` so two recipes' deps of the same kind do + not collide on a single node). +3. Waits each dep healthy using its own `recipe_meta.py` (HEALTH_PATH/HEALTH_OK/timeouts). +4. Persists `[{"recipe": "", "domain": ""}, ...]` to `$CCCI_DEPS_FILE`. +5. Deploys + tests the recipe under test as usual. +6. Tears down the dep LAST in `finally` (reverse declaration order, with `verify=True` — leaked + deps fail the run loudly per §9 teardown sacred / F2-5 fix). + +Tests access dep domains via the **`deps_apps` pytest fixture** (`tests/conftest.py`): + +```python +def test_my_recipe_uses_keycloak(live_app, deps_apps): + assert "keycloak" in deps_apps, f"keycloak dep not deployed; {deps_apps}" + kc_domain = deps_apps["keycloak"] + … +``` + +Deploy-count guard: with deps the expected count is `1 + len(DEPS)` (the parent + one per dep). +The orchestrator computes this and fails the run on mismatch. + +### 2.3 SSO setup — harness.sso (Phase 2 Q2.3) + +For OIDC-dependent recipes, the shared `runner/harness/sso.py` provides: + +```python +from harness import sso + +creds = sso.setup_keycloak_realm( + kc_domain, # = deps_apps["keycloak"] + realm="my-realm", + client_id="my-client", + redirect_uris=[f"https://{live_app}/*"], + web_origins=[f"https://{live_app}"], +) +# creds = {"realm", "client_id", "client_secret", "user", "password", "token_url", …} + +sso.assert_discovery_endpoint(creds) # GET /.well-known/openid-configuration +token = sso.oidc_password_grant(creds) # exercises the OIDC password grant; returns JWT +``` + +`setup_keycloak_realm` is **idempotent** (409 → reset to known values) and uses **class-B +run-scoped secrets** (the generated `client_secret` + test-user password are destroyed when the +dep keycloak is torn down at run end, plan §4.4-B). **Note (F2-7):** the setup primitive is +keycloak-specific; when authentik comes online a parallel `setup_authentik_realm` will need to +land in `harness.sso`. The flow primitives (`oidc_password_grant`, `assert_discovery_endpoint`) +ARE provider-pluggable. + ## 3. Recipe-local tests (D4) — default-deny (HC2) If the recipe's own repo contains `tests/test_*.py` / `install_steps.sh` / `ops.py`, the runner @@ -89,5 +174,33 @@ The webhook and poller are deduped by comment id, so a comment seen by both fire ```sh RECIPE= PR= REF= SRC=recipe-maintainers/ \ - STAGES=install,upgrade,backup cc-ci-run runner/run_recipe_ci.py + STAGES=install,upgrade,backup,restore,custom cc-ci-run runner/run_recipe_ci.py ``` + +## Worked example — lasuite-docs (OIDC-dependent, Phase 2) + +``` +tests/lasuite-docs/ +├── recipe_meta.py # HEALTH_PATH="/", DEPLOY_TIMEOUT=900, EXTRA_ENV(domain) for cold-pull, +│ # DEPS=["keycloak"] ← Phase 2 dep declaration +├── ops.py # pre_ seed hooks (volume marker for backup/restore data-integrity) +├── test_install.py # lifecycle install overlay (Playwright frontend SPA load) +├── test_upgrade.py # lifecycle upgrade overlay (marker survives chaos redeploy) +├── test_backup.py # lifecycle backup overlay (marker captured) +├── test_restore.py # lifecycle restore overlay (marker restored to pre-mutation) +├── PARITY.md # parity-port mapping (P2) +└── functional/ + ├── test_health_check.py # parity port (SOURCE comment cites recipe-info file) + ├── test_auth_required.py # specific: /api/v1.0/users/me/ → 401 without auth + └── test_oidc_with_keycloak.py # specific: full OIDC flow against the dep keycloak (uses + # harness.sso primitives + deps_apps["keycloak"]) +``` + +`!testme` on a lasuite-docs PR drives the orchestrator to: +1. Deploy the per-run keycloak dep (`keyc-<6hex>.ci.commoninternet.net`) and wait healthy. +2. Deploy lasuite-docs (`lasu-<6hex>.ci.commoninternet.net`). +3. Run install / upgrade / backup / restore + the 3 functional tests against the shared + deployment (custom tier). +4. Teardown lasuite-docs, then the keycloak dep (LAST), both with verify=True. +5. Print the run summary; non-zero exit code on any failure (DG4.1 deploy-count mismatch, tier + FAIL, dep teardown leak — all surfaced).