feat(cfold): canonicalize custom test layout
Some checks failed
continuous-integration/drone/push Build is failing
Some checks failed
continuous-integration/drone/push Build is failing
This commit is contained in:
@ -22,7 +22,7 @@ A recipe customizes its CI through **three distinct mechanisms**:
|
||||
|---|---|---|
|
||||
| **Declarative settings** | Python assignments in `tests/<recipe>/recipe_meta.py` | `DEPLOY_TIMEOUT = 1500`, `UPGRADE_BASE_VERSION = "2.3.1+..."` |
|
||||
| **Code hooks** | Callables in `recipe_meta.py`, `ops.py` functions, one shell hook | `def READY_PROBE(ctx): ...`, `pre_upgrade(ctx)`, `install_steps.sh` |
|
||||
| **File presence** | A file existing at a discovered path changes behavior | `test_upgrade.py` overlay, `functional/test_*.py`, `compose.ccci.yml` |
|
||||
| **File presence** | A file existing at a discovered path changes behavior | `test_upgrade.py` overlay, `custom/test_*.py`, `compose.ccci.yml` |
|
||||
|
||||
There is additionally a fourth, **operator-facing, local-dev-only** surface: environment variables
|
||||
(`CCCI_SKIP_GENERIC*`) that suppress the generic floor at run time (§7). Whatever a run resolves
|
||||
@ -60,15 +60,15 @@ tests/<recipe>/ # cc-ci side (repo-local mirrors the same s
|
||||
├── recipe_meta.py # THE config file: registry-validated keys + ctx-hooks (§4)
|
||||
├── test_<op>.py # lifecycle overlay assertions, op ∈ install|upgrade|backup|restore (§5.1)
|
||||
├── ops.py # pre_<op>(ctx) seed hooks (§5.2)
|
||||
├── functional/test_*.py # custom tier: parity ports + recipe-specific (§5.3)
|
||||
├── playwright/test_*.py # custom tier: UI flows (§5.3)
|
||||
├── custom/test_*.py # custom tier: parity ports + recipe-specific + UI flows (§5.3)
|
||||
├── install_steps.sh # pre-deploy shell hook (the ONLY shell hook) (§5.4)
|
||||
├── compose.ccci.yml # CI-only compose overlay (first-class) (§5.5)
|
||||
└── PARITY.md # enrollment contract doc (human-read only)
|
||||
```
|
||||
|
||||
**Placement rule (custom tests):** ALL custom-tier tests live under `functional/` or
|
||||
`playwright/`. A top-level `test_*.py` is a lifecycle overlay (`test_<op>.py`) and nothing else —
|
||||
**Placement rule (custom tests):** ALL custom-tier tests live under canonical `custom/`.
|
||||
Deprecated `functional/` and `playwright/` aliases are still discovered with a loud warning so
|
||||
coverage is not silently lost while recipe trees migrate. A top-level `test_*.py` is a lifecycle overlay (`test_<op>.py`) and nothing else —
|
||||
top-level non-lifecycle files are NOT discovered (`discovery.custom_tests`; the lifecycle-name
|
||||
exclusion stays as a safety net so a misfiled `test_<op>.py` can never double-run).
|
||||
|
||||
@ -76,7 +76,8 @@ Precedence (machine-docs/DECISIONS.md, implemented in `discovery.py`):
|
||||
|
||||
- lifecycle overlay `test_<op>.py`: repo-local **wins** over cc-ci (same-name collision); the
|
||||
generic floor still runs additively alongside.
|
||||
- custom tier (`functional/` + `playwright/`): **ALL** run, from both locations (no collision
|
||||
- custom tier (`custom/`, plus deprecated alias dirs during migration): **ALL** run, from both
|
||||
locations (no collision
|
||||
concept).
|
||||
- `install_steps.sh`: repo-local > cc-ci, or none.
|
||||
- `ops.py` pre-op hook: cc-ci wins; repo-local consulted only if approved.
|
||||
@ -181,15 +182,16 @@ def pre_restore(ctx): _psql(ctx.domain, "DROP TABLE ci_marker") # damage, rest
|
||||
Seed → op → assert is the whole pattern: `pre_backup` writes a marker, the orchestrator backs up,
|
||||
`pre_restore` destroys it, the orchestrator restores, `test_restore.py` asserts the marker is back.
|
||||
|
||||
### 5.3 Custom tier — `functional/` and `playwright/` ONLY
|
||||
### 5.3 Custom tier — canonical `custom/`
|
||||
|
||||
All custom-tier tests live under `tests/<recipe>/functional/` or `tests/<recipe>/playwright/`
|
||||
(discovery: `discovery.custom_tests`; the placement rule, §3). Run in the CUSTOM tier, after
|
||||
All custom-tier tests live under `tests/<recipe>/custom/` (discovery: `discovery.custom_tests`;
|
||||
the placement rule, §3). Deprecated `functional/` and `playwright/` dirs are still recognized
|
||||
with a warning during the migration window. Custom tests run in the CUSTOM tier, after
|
||||
restore, against the post-upgrade (PR-head) app. ALL discovered files run — cc-ci's and (if
|
||||
HC2-approved) repo-local's, additively.
|
||||
|
||||
Enrollment contract (`docs/enroll-recipe.md`): ≥2 NEW functional tests beyond ports of existing
|
||||
upstream checks; ported tests carry `SOURCE:` comments. Playwright tests get the shared
|
||||
Enrollment contract (`docs/enroll-recipe.md`): ≥2 NEW custom tests beyond ports of existing
|
||||
upstream checks; ported tests carry `SOURCE:` comments. Browser-driven custom tests get the shared
|
||||
browser/harness helpers (`harness.browser`); SSO recipes get `harness.sso`
|
||||
(`setup_keycloak_realm` — idempotent, `oidc_password_grant` — provider-pluggable). The documented
|
||||
import toolbox for custom tests is `from harness import lifecycle, sso, browser`.
|
||||
@ -268,7 +270,7 @@ deploy BASE (UPGRADE_BASE_VERSION or recipe_versions[-2]; EXTRA_ENV; install_ste
|
||||
→ BACKUP tier
|
||||
→ pre_restore(ctx) → restore
|
||||
→ RESTORE tier
|
||||
→ CUSTOM tier (functional/ + playwright/; deps via the `deps` fixture)
|
||||
→ CUSTOM tier (custom/; deps via the `deps` fixture)
|
||||
→ SCREENSHOT (best-effort, never affects the verdict)
|
||||
→ teardown (deps LAST)
|
||||
```
|
||||
@ -293,7 +295,7 @@ RECIPE=<recipe> PR=<n> REF=<sha> SRC=recipe-maintainers/<recipe> \
|
||||
meta (non-default): DEPLOY_TIMEOUT=1500 DEPS=['keycloak'] EXTRA_ENV='<hook>'
|
||||
hooks: ops.py[pre_backup,pre_upgrade](cc-ci) install_steps.sh(cc-ci) compose.ccci.yml(cc-ci)
|
||||
overlays: test_backup.py(cc-ci) test_restore.py(repo-local)
|
||||
custom tests: functional/=5 playwright/=2 (cc-ci)
|
||||
custom tests: custom/=7 (cc-ci)
|
||||
env overrides: (none)
|
||||
```
|
||||
|
||||
|
||||
Reference in New Issue
Block a user