feat(cfold): canonicalize custom test layout
Some checks failed
continuous-integration/drone/push Build is failing

This commit is contained in:
autonomic-bot
2026-06-12 16:08:18 +00:00
parent 87928a9096
commit 44e02425ab
110 changed files with 306 additions and 241 deletions

View File

@ -22,7 +22,7 @@ A recipe customizes its CI through **three distinct mechanisms**:
|---|---|---|
| **Declarative settings** | Python assignments in `tests/<recipe>/recipe_meta.py` | `DEPLOY_TIMEOUT = 1500`, `UPGRADE_BASE_VERSION = "2.3.1+..."` |
| **Code hooks** | Callables in `recipe_meta.py`, `ops.py` functions, one shell hook | `def READY_PROBE(ctx): ...`, `pre_upgrade(ctx)`, `install_steps.sh` |
| **File presence** | A file existing at a discovered path changes behavior | `test_upgrade.py` overlay, `functional/test_*.py`, `compose.ccci.yml` |
| **File presence** | A file existing at a discovered path changes behavior | `test_upgrade.py` overlay, `custom/test_*.py`, `compose.ccci.yml` |
There is additionally a fourth, **operator-facing, local-dev-only** surface: environment variables
(`CCCI_SKIP_GENERIC*`) that suppress the generic floor at run time (§7). Whatever a run resolves
@ -60,15 +60,15 @@ tests/<recipe>/ # cc-ci side (repo-local mirrors the same s
├── recipe_meta.py # THE config file: registry-validated keys + ctx-hooks (§4)
├── test_<op>.py # lifecycle overlay assertions, op ∈ install|upgrade|backup|restore (§5.1)
├── ops.py # pre_<op>(ctx) seed hooks (§5.2)
├── functional/test_*.py # custom tier: parity ports + recipe-specific (§5.3)
├── playwright/test_*.py # custom tier: UI flows (§5.3)
├── custom/test_*.py # custom tier: parity ports + recipe-specific + UI flows (§5.3)
├── install_steps.sh # pre-deploy shell hook (the ONLY shell hook) (§5.4)
├── compose.ccci.yml # CI-only compose overlay (first-class) (§5.5)
└── PARITY.md # enrollment contract doc (human-read only)
```
**Placement rule (custom tests):** ALL custom-tier tests live under `functional/` or
`playwright/`. A top-level `test_*.py` is a lifecycle overlay (`test_<op>.py`) and nothing else —
**Placement rule (custom tests):** ALL custom-tier tests live under canonical `custom/`.
Deprecated `functional/` and `playwright/` aliases are still discovered with a loud warning so
coverage is not silently lost while recipe trees migrate. A top-level `test_*.py` is a lifecycle overlay (`test_<op>.py`) and nothing else —
top-level non-lifecycle files are NOT discovered (`discovery.custom_tests`; the lifecycle-name
exclusion stays as a safety net so a misfiled `test_<op>.py` can never double-run).
@ -76,7 +76,8 @@ Precedence (machine-docs/DECISIONS.md, implemented in `discovery.py`):
- lifecycle overlay `test_<op>.py`: repo-local **wins** over cc-ci (same-name collision); the
generic floor still runs additively alongside.
- custom tier (`functional/` + `playwright/`): **ALL** run, from both locations (no collision
- custom tier (`custom/`, plus deprecated alias dirs during migration): **ALL** run, from both
locations (no collision
concept).
- `install_steps.sh`: repo-local > cc-ci, or none.
- `ops.py` pre-op hook: cc-ci wins; repo-local consulted only if approved.
@ -181,15 +182,16 @@ def pre_restore(ctx): _psql(ctx.domain, "DROP TABLE ci_marker") # damage, rest
Seed → op → assert is the whole pattern: `pre_backup` writes a marker, the orchestrator backs up,
`pre_restore` destroys it, the orchestrator restores, `test_restore.py` asserts the marker is back.
### 5.3 Custom tier — `functional/` and `playwright/` ONLY
### 5.3 Custom tier — canonical `custom/`
All custom-tier tests live under `tests/<recipe>/functional/` or `tests/<recipe>/playwright/`
(discovery: `discovery.custom_tests`; the placement rule, §3). Run in the CUSTOM tier, after
All custom-tier tests live under `tests/<recipe>/custom/` (discovery: `discovery.custom_tests`;
the placement rule, §3). Deprecated `functional/` and `playwright/` dirs are still recognized
with a warning during the migration window. Custom tests run in the CUSTOM tier, after
restore, against the post-upgrade (PR-head) app. ALL discovered files run — cc-ci's and (if
HC2-approved) repo-local's, additively.
Enrollment contract (`docs/enroll-recipe.md`): ≥2 NEW functional tests beyond ports of existing
upstream checks; ported tests carry `SOURCE:` comments. Playwright tests get the shared
Enrollment contract (`docs/enroll-recipe.md`): ≥2 NEW custom tests beyond ports of existing
upstream checks; ported tests carry `SOURCE:` comments. Browser-driven custom tests get the shared
browser/harness helpers (`harness.browser`); SSO recipes get `harness.sso`
(`setup_keycloak_realm` — idempotent, `oidc_password_grant` — provider-pluggable). The documented
import toolbox for custom tests is `from harness import lifecycle, sso, browser`.
@ -268,7 +270,7 @@ deploy BASE (UPGRADE_BASE_VERSION or recipe_versions[-2]; EXTRA_ENV; install_ste
→ BACKUP tier
→ pre_restore(ctx) → restore
→ RESTORE tier
→ CUSTOM tier (functional/ + playwright/; deps via the `deps` fixture)
→ CUSTOM tier (custom/; deps via the `deps` fixture)
→ SCREENSHOT (best-effort, never affects the verdict)
→ teardown (deps LAST)
```
@ -293,7 +295,7 @@ RECIPE=<recipe> PR=<n> REF=<sha> SRC=recipe-maintainers/<recipe> \
meta (non-default): DEPLOY_TIMEOUT=1500 DEPS=['keycloak'] EXTRA_ENV='<hook>'
hooks: ops.py[pre_backup,pre_upgrade](cc-ci) install_steps.sh(cc-ci) compose.ccci.yml(cc-ci)
overlays: test_backup.py(cc-ci) test_restore.py(repo-local)
custom tests: functional/=5 playwright/=2 (cc-ci)
custom tests: custom/=7 (cc-ci)
env overrides: (none)
```