docs(2): Q5.1 partial — enroll-recipe.md Phase-2 contract
Adds: - §2 layout: PARITY.md / functional/ / playwright/ subdirs (Phase 2 §4.1) - §2.1 Phase-2 contract: parity port + ≥2 specific functional tests + Playwright; custom-tier discovery from functional/ + playwright/; SOURCE comment audit - §2.2 DEPS = [...] declaration; orchestrator dep deploy order; deps_apps fixture; expected deploy-count = 1 + len(DEPS); F2-5 verify=True teardown - §2.3 harness.sso primitives (setup_keycloak_realm, oidc_password_grant, assert_discovery_endpoint); F2-7 note that setup is keycloak-specific - Worked example: lasuite-docs full Phase-2 layout (DEPS + functional/ + lifecycle overlays) and the !testme flow walked through end-to-end - Updated 'Run locally' to include restore + custom stages A new engineer can add a recipe's full Phase-2 suite from the docs alone (P8). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
@ -19,7 +19,14 @@ tests/<recipe>/
|
||||
├── test_install.py # optional install overlay (runs ADDITIVELY alongside generic)
|
||||
├── test_upgrade.py # optional upgrade overlay (runs ADDITIVELY alongside generic)
|
||||
├── test_backup.py # optional backup overlay (runs ADDITIVELY alongside generic)
|
||||
└── test_restore.py # optional restore overlay (runs ADDITIVELY alongside generic)
|
||||
├── test_restore.py # optional restore overlay (runs ADDITIVELY alongside generic)
|
||||
├── PARITY.md # Phase 2 P2: mapping table (recipe-maintainer tests → cc-ci tests)
|
||||
├── functional/ # Phase 2 P3: parity ports + ≥2 NEW recipe-specific tests
|
||||
│ ├── test_health_check.py # parity port of recipe-info/<recipe>/tests/health_check.py
|
||||
│ ├── test_<behavior>.py # ≥2 NEW recipe-specific functional tests
|
||||
│ └── …
|
||||
└── playwright/ # Phase 2 P6: browser flows where the app's core UX is a UI
|
||||
└── test_<flow>.py
|
||||
```
|
||||
|
||||
**A recipe is testable with ZERO config:** with no overlay files, the **generic lifecycle suite**
|
||||
@ -54,6 +61,84 @@ Useful `harness.lifecycle` helpers for overlays: `http_get`, `http_fetch`, `http
|
||||
ops themselves are orchestrator-owned (you never call them from an overlay). The harness forces
|
||||
`LETS_ENCRYPT_ENV=""` (no ACME), a unique short domain per run, and guarantees teardown.
|
||||
|
||||
### 2.1 Phase-2 contract: parity port + recipe-specific functional tests + Playwright
|
||||
|
||||
Beyond the lifecycle overlays, each recipe carries (plan §4.1):
|
||||
|
||||
- **`PARITY.md`** — a mapping table from every `references/recipe-maintainer/recipe-info/<recipe>/
|
||||
tests/*.py` to a comparable cc-ci test under `tests/<recipe>/functional/`, asserting the
|
||||
*same thing* (not a renamed file). A deliberate non-port is documented in `DECISIONS.md` with
|
||||
a technical reason — never a silent omission.
|
||||
- **`functional/`** — parity-port tests + **≥2 NEW recipe-specific functional tests** that
|
||||
exercise the app's characteristic behavior (per plan §4.3 — e.g. "create-an-object +
|
||||
read-it-back, and one more that touches a distinctive feature"). Each parity-port file carries
|
||||
a `SOURCE = "recipe-info/<recipe>/tests/<file>"` comment near the top so audit is in-file.
|
||||
- **`playwright/`** — browser flows where the recipe's core UX is a UI (P6).
|
||||
|
||||
The orchestrator's **custom** tier discovers `test_*.py` in `tests/<recipe>/{functional,playwright}/`
|
||||
(recursive, via `runner/harness/discovery.custom_tests`) and runs each as its own pytest against
|
||||
the same `live_app` shared deployment. Lifecycle-named files (`test_install.py`/etc.) are
|
||||
**excluded** from the custom tier — they live at the top level and run as lifecycle overlays.
|
||||
|
||||
### 2.2 Recipe-test dependencies — DEPS = [...] (Phase 2 Q2.3)
|
||||
|
||||
If your recipe needs other recipes deployed alongside it (an SSO provider, a database), declare
|
||||
them in `recipe_meta.py`:
|
||||
|
||||
```python
|
||||
DEPS = ["keycloak"] # one entry per dep recipe name (cc-ci tests/<dep>/ must exist + work)
|
||||
```
|
||||
|
||||
The orchestrator (plan §4.2):
|
||||
1. Reads `DEPS` BEFORE deploying the recipe under test.
|
||||
2. Deploys each dep at a per-run domain `<dep[:4]>-<6hex>.ci.commoninternet.net` (the 6hex is
|
||||
hashed from `parent_recipe + pr + ref + dep_recipe` so two recipes' deps of the same kind do
|
||||
not collide on a single node).
|
||||
3. Waits each dep healthy using its own `recipe_meta.py` (HEALTH_PATH/HEALTH_OK/timeouts).
|
||||
4. Persists `[{"recipe": "<dep>", "domain": "<dep-domain>"}, ...]` to `$CCCI_DEPS_FILE`.
|
||||
5. Deploys + tests the recipe under test as usual.
|
||||
6. Tears down the dep LAST in `finally` (reverse declaration order, with `verify=True` — leaked
|
||||
deps fail the run loudly per §9 teardown sacred / F2-5 fix).
|
||||
|
||||
Tests access dep domains via the **`deps_apps` pytest fixture** (`tests/conftest.py`):
|
||||
|
||||
```python
|
||||
def test_my_recipe_uses_keycloak(live_app, deps_apps):
|
||||
assert "keycloak" in deps_apps, f"keycloak dep not deployed; {deps_apps}"
|
||||
kc_domain = deps_apps["keycloak"]
|
||||
…
|
||||
```
|
||||
|
||||
Deploy-count guard: with deps the expected count is `1 + len(DEPS)` (the parent + one per dep).
|
||||
The orchestrator computes this and fails the run on mismatch.
|
||||
|
||||
### 2.3 SSO setup — harness.sso (Phase 2 Q2.3)
|
||||
|
||||
For OIDC-dependent recipes, the shared `runner/harness/sso.py` provides:
|
||||
|
||||
```python
|
||||
from harness import sso
|
||||
|
||||
creds = sso.setup_keycloak_realm(
|
||||
kc_domain, # = deps_apps["keycloak"]
|
||||
realm="my-realm",
|
||||
client_id="my-client",
|
||||
redirect_uris=[f"https://{live_app}/*"],
|
||||
web_origins=[f"https://{live_app}"],
|
||||
)
|
||||
# creds = {"realm", "client_id", "client_secret", "user", "password", "token_url", …}
|
||||
|
||||
sso.assert_discovery_endpoint(creds) # GET /.well-known/openid-configuration
|
||||
token = sso.oidc_password_grant(creds) # exercises the OIDC password grant; returns JWT
|
||||
```
|
||||
|
||||
`setup_keycloak_realm` is **idempotent** (409 → reset to known values) and uses **class-B
|
||||
run-scoped secrets** (the generated `client_secret` + test-user password are destroyed when the
|
||||
dep keycloak is torn down at run end, plan §4.4-B). **Note (F2-7):** the setup primitive is
|
||||
keycloak-specific; when authentik comes online a parallel `setup_authentik_realm` will need to
|
||||
land in `harness.sso`. The flow primitives (`oidc_password_grant`, `assert_discovery_endpoint`)
|
||||
ARE provider-pluggable.
|
||||
|
||||
## 3. Recipe-local tests (D4) — default-deny (HC2)
|
||||
|
||||
If the recipe's own repo contains `tests/test_*.py` / `install_steps.sh` / `ops.py`, the runner
|
||||
@ -89,5 +174,33 @@ The webhook and poller are deduped by comment id, so a comment seen by both fire
|
||||
|
||||
```sh
|
||||
RECIPE=<recipe> PR=<n> REF=<sha-or-branch> SRC=recipe-maintainers/<recipe> \
|
||||
STAGES=install,upgrade,backup cc-ci-run runner/run_recipe_ci.py
|
||||
STAGES=install,upgrade,backup,restore,custom cc-ci-run runner/run_recipe_ci.py
|
||||
```
|
||||
|
||||
## Worked example — lasuite-docs (OIDC-dependent, Phase 2)
|
||||
|
||||
```
|
||||
tests/lasuite-docs/
|
||||
├── recipe_meta.py # HEALTH_PATH="/", DEPLOY_TIMEOUT=900, EXTRA_ENV(domain) for cold-pull,
|
||||
│ # DEPS=["keycloak"] ← Phase 2 dep declaration
|
||||
├── ops.py # pre_<op> seed hooks (volume marker for backup/restore data-integrity)
|
||||
├── test_install.py # lifecycle install overlay (Playwright frontend SPA load)
|
||||
├── test_upgrade.py # lifecycle upgrade overlay (marker survives chaos redeploy)
|
||||
├── test_backup.py # lifecycle backup overlay (marker captured)
|
||||
├── test_restore.py # lifecycle restore overlay (marker restored to pre-mutation)
|
||||
├── PARITY.md # parity-port mapping (P2)
|
||||
└── functional/
|
||||
├── test_health_check.py # parity port (SOURCE comment cites recipe-info file)
|
||||
├── test_auth_required.py # specific: /api/v1.0/users/me/ → 401 without auth
|
||||
└── test_oidc_with_keycloak.py # specific: full OIDC flow against the dep keycloak (uses
|
||||
# harness.sso primitives + deps_apps["keycloak"])
|
||||
```
|
||||
|
||||
`!testme` on a lasuite-docs PR drives the orchestrator to:
|
||||
1. Deploy the per-run keycloak dep (`keyc-<6hex>.ci.commoninternet.net`) and wait healthy.
|
||||
2. Deploy lasuite-docs (`lasu-<6hex>.ci.commoninternet.net`).
|
||||
3. Run install / upgrade / backup / restore + the 3 functional tests against the shared
|
||||
deployment (custom tier).
|
||||
4. Teardown lasuite-docs, then the keycloak dep (LAST), both with verify=True.
|
||||
5. Print the run summary; non-zero exit code on any failure (DG4.1 deploy-count mismatch, tier
|
||||
FAIL, dep teardown leak — all surfaced).
|
||||
|
||||
Reference in New Issue
Block a user