feat(cfold): canonicalize custom test layout
Some checks failed
continuous-integration/drone/push Build is failing
Some checks failed
continuous-integration/drone/push Build is failing
This commit is contained in:
autonomic-bot
@ -22,12 +22,11 @@ tests/<recipe>/
|
||||
├── test_backup.py # optional backup overlay (runs ADDITIVELY alongside generic)
|
||||
├── test_restore.py # optional restore overlay (runs ADDITIVELY alongside generic)
|
||||
├── PARITY.md # Phase 2 P2: mapping table (recipe-maintainer tests → cc-ci tests)
|
||||
├── functional/ # Phase 2 P3: parity ports + ≥2 NEW recipe-specific tests
|
||||
│ ├── test_health_check.py # parity port of recipe-info/<recipe>/tests/health_check.py
|
||||
│ ├── test_<behavior>.py # ≥2 NEW recipe-specific functional tests
|
||||
│ └── …
|
||||
└── playwright/ # Phase 2 P6: browser flows where the app's core UX is a UI
|
||||
└── test_<flow>.py
|
||||
└── custom/ # custom tier: parity ports + recipe-specific tests + browser flows
|
||||
├── test_health_check.py # parity port of recipe-info/<recipe>/tests/health_check.py
|
||||
├── test_<behavior>.py # ≥2 NEW recipe-specific tests
|
||||
├── test_<flow>.py # browser/UI flows where relevant
|
||||
└── …
|
||||
```
|
||||
|
||||
**A recipe is testable with ZERO config:** with no overlay files, the **generic lifecycle suite**
|
||||
@ -68,18 +67,18 @@ ops themselves are orchestrator-owned (you never call them from an overlay). The
|
||||
Beyond the lifecycle overlays, each recipe carries (plan §4.1):
|
||||
|
||||
- **`PARITY.md`** — a mapping table from every `references/recipe-maintainer/recipe-info/<recipe>/
|
||||
tests/*.py` to a comparable cc-ci test under `tests/<recipe>/functional/`, asserting the
|
||||
tests/*.py` to a comparable cc-ci test under `tests/<recipe>/custom/`, asserting the
|
||||
*same thing* (not a renamed file). A deliberate non-port is documented in `DECISIONS.md` with
|
||||
a technical reason — never a silent omission.
|
||||
- **`functional/`** — parity-port tests + **≥2 NEW recipe-specific functional tests** that
|
||||
exercise the app's characteristic behavior (per plan §4.3 — e.g. "create-an-object +
|
||||
read-it-back, and one more that touches a distinctive feature"). Each parity-port file carries
|
||||
a `SOURCE = "recipe-info/<recipe>/tests/<file>"` comment near the top so audit is in-file.
|
||||
- **`playwright/`** — browser flows where the recipe's core UX is a UI (P6).
|
||||
- **`custom/`** — parity-port tests + **≥2 NEW recipe-specific tests** that exercise the app's
|
||||
characteristic behavior (per plan §4.3 — e.g. "create-an-object + read-it-back, and one more
|
||||
that touches a distinctive feature"). Browser/UI flows live in the same folder too. Each
|
||||
parity-port file carries a `SOURCE = "recipe-info/<recipe>/tests/<file>"` comment near the top
|
||||
so audit is in-file.
|
||||
|
||||
The orchestrator's **custom** tier discovers `test_*.py` in `tests/<recipe>/{functional,playwright}/`
|
||||
ONLY (the placement rule, via `runner/harness/discovery.custom_tests` — a top-level `test_*.py`
|
||||
is a lifecycle overlay and nothing else) and runs each as its own pytest against the same
|
||||
The orchestrator's **custom** tier discovers `test_*.py` in canonical `tests/<recipe>/custom/`
|
||||
(plus deprecated `functional/` / `playwright/` aliases during migration; discovery warns when it
|
||||
uses them) and runs each as its own pytest against the same
|
||||
`live_app` shared deployment. Lifecycle-named files (`test_install.py`/etc.) are **excluded**
|
||||
from the custom tier even inside those subdirs (safety net against double-running).
|
||||
|
||||
@ -176,7 +175,7 @@ shapes (proven on mumble, mailu, and the SSO-dependent suite):
|
||||
|
||||
**Non-HTTP protocol tests (mumble).** Reach a TCP service published `mode: host` (via a host-ports
|
||||
overlay) at `127.0.0.1:<port>` — cc-ci runs tests on-host (cc-ci-run). mumble ships a stdlib protocol
|
||||
client (`tests/mumble/functional/_mumble_proto.py`) doing the real TLS handshake → ServerSync; the
|
||||
client (`tests/mumble/custom/_mumble_proto.py`) doing the real TLS handshake → ServerSync; the
|
||||
recipe-specific tests assert channel presence and config round-trips (a deploy-set `WELCOME_TEXT`/
|
||||
`USERS` value surfaces over the protocol — version-independent, non-vacuous).
|
||||
|
||||
@ -244,7 +243,7 @@ tests/lasuite-docs/
|
||||
├── test_backup.py # lifecycle backup overlay (marker captured)
|
||||
├── test_restore.py # lifecycle restore overlay (marker restored to pre-mutation)
|
||||
├── PARITY.md # parity-port mapping (P2)
|
||||
└── functional/
|
||||
└── custom/
|
||||
├── test_health_check.py # parity port (SOURCE comment cites recipe-info file)
|
||||
├── test_auth_required.py # specific: /api/v1.0/users/me/ → 401 without auth
|
||||
└── test_oidc_with_keycloak.py # specific: full OIDC flow against the dep keycloak (uses
|
||||
@ -256,8 +255,8 @@ tests/lasuite-docs/
|
||||
creds to `$CCCI_DEPS_FILE` — BEFORE the recipe deploy.
|
||||
2. Deploy lasuite-docs (`lasu-<6hex>.ci.commoninternet.net`); `install_steps.sh` wires the OIDC
|
||||
env into that one deploy.
|
||||
3. Run install / upgrade / backup / restore + the 3 functional tests against the shared
|
||||
deployment (custom tier).
|
||||
3. Run install / upgrade / backup / restore + the 3 custom tests against the shared
|
||||
deployment (custom tier).
|
||||
4. Teardown lasuite-docs, then the keycloak dep (LAST), both with verify=True.
|
||||
5. Print the run summary; non-zero exit code on any failure (DG4.1 deploy-count mismatch, tier
|
||||
FAIL, dep teardown leak — all surfaced).
|
||||
@ -268,10 +267,10 @@ tests/lasuite-docs/
|
||||
`COMPOSE_FILE=compose.yml:compose.mumbleweb.yml` for the base; `UPGRADE_EXTRA_ENV` adds the
|
||||
native `compose.host-ports.yml` at PR-head so 64738 is host-published on latest; private
|
||||
`_WELCOME_TEXT_MARKER`/`_MAX_USERS` constants; `READY_PROBE(ctx)` TCP 64738 — phase-aware via
|
||||
the live COMPOSE_FILE), `functional/_mumble_proto.py` + the protocol/config-round-trip
|
||||
the live COMPOSE_FILE), `custom/_mumble_proto.py` + the protocol/config-round-trip
|
||||
tests, `ops.py`/`test_backup.py`/`test_restore.py` (sqlite P4). See §2.4.
|
||||
- **Multi-service, dep-less, in-container functional — `tests/mailu/`**: `recipe_meta.py`
|
||||
(`EXTRA_ENV(ctx)` with `TLS_FLAVOR=notls` + `MAIL_DOMAIN`/`HOSTNAMES`/`TRAEFIK_STACK_NAME`),
|
||||
`functional/_mailu.py` (flask-CLI helpers), `test_mailbox.py` (create→config-export read-back),
|
||||
`custom/_mailu.py` (flask-CLI helpers), `test_mailbox.py` (create→config-export read-back),
|
||||
`test_mail_flow.py` (in-container sendmail→doveadm delivery). No backupbot → P4 N/A (PARITY.md +
|
||||
DEFERRED.md). See §2.4.
|
||||
|
||||
@ -22,7 +22,7 @@ A recipe customizes its CI through **three distinct mechanisms**:
|
||||
|---|---|---|
|
||||
| **Declarative settings** | Python assignments in `tests/<recipe>/recipe_meta.py` | `DEPLOY_TIMEOUT = 1500`, `UPGRADE_BASE_VERSION = "2.3.1+..."` |
|
||||
| **Code hooks** | Callables in `recipe_meta.py`, `ops.py` functions, one shell hook | `def READY_PROBE(ctx): ...`, `pre_upgrade(ctx)`, `install_steps.sh` |
|
||||
| **File presence** | A file existing at a discovered path changes behavior | `test_upgrade.py` overlay, `functional/test_*.py`, `compose.ccci.yml` |
|
||||
| **File presence** | A file existing at a discovered path changes behavior | `test_upgrade.py` overlay, `custom/test_*.py`, `compose.ccci.yml` |
|
||||
|
||||
There is additionally a fourth, **operator-facing, local-dev-only** surface: environment variables
|
||||
(`CCCI_SKIP_GENERIC*`) that suppress the generic floor at run time (§7). Whatever a run resolves
|
||||
@ -60,15 +60,15 @@ tests/<recipe>/ # cc-ci side (repo-local mirrors the same s
|
||||
├── recipe_meta.py # THE config file: registry-validated keys + ctx-hooks (§4)
|
||||
├── test_<op>.py # lifecycle overlay assertions, op ∈ install|upgrade|backup|restore (§5.1)
|
||||
├── ops.py # pre_<op>(ctx) seed hooks (§5.2)
|
||||
├── functional/test_*.py # custom tier: parity ports + recipe-specific (§5.3)
|
||||
├── playwright/test_*.py # custom tier: UI flows (§5.3)
|
||||
├── custom/test_*.py # custom tier: parity ports + recipe-specific + UI flows (§5.3)
|
||||
├── install_steps.sh # pre-deploy shell hook (the ONLY shell hook) (§5.4)
|
||||
├── compose.ccci.yml # CI-only compose overlay (first-class) (§5.5)
|
||||
└── PARITY.md # enrollment contract doc (human-read only)
|
||||
```
|
||||
|
||||
**Placement rule (custom tests):** ALL custom-tier tests live under `functional/` or
|
||||
`playwright/`. A top-level `test_*.py` is a lifecycle overlay (`test_<op>.py`) and nothing else —
|
||||
**Placement rule (custom tests):** ALL custom-tier tests live under canonical `custom/`.
|
||||
Deprecated `functional/` and `playwright/` aliases are still discovered with a loud warning so
|
||||
coverage is not silently lost while recipe trees migrate. A top-level `test_*.py` is a lifecycle overlay (`test_<op>.py`) and nothing else —
|
||||
top-level non-lifecycle files are NOT discovered (`discovery.custom_tests`; the lifecycle-name
|
||||
exclusion stays as a safety net so a misfiled `test_<op>.py` can never double-run).
|
||||
|
||||
@ -76,7 +76,8 @@ Precedence (machine-docs/DECISIONS.md, implemented in `discovery.py`):
|
||||
|
||||
- lifecycle overlay `test_<op>.py`: repo-local **wins** over cc-ci (same-name collision); the
|
||||
generic floor still runs additively alongside.
|
||||
- custom tier (`functional/` + `playwright/`): **ALL** run, from both locations (no collision
|
||||
- custom tier (`custom/`, plus deprecated alias dirs during migration): **ALL** run, from both
|
||||
locations (no collision
|
||||
concept).
|
||||
- `install_steps.sh`: repo-local > cc-ci, or none.
|
||||
- `ops.py` pre-op hook: cc-ci wins; repo-local consulted only if approved.
|
||||
@ -181,15 +182,16 @@ def pre_restore(ctx): _psql(ctx.domain, "DROP TABLE ci_marker") # damage, rest
|
||||
Seed → op → assert is the whole pattern: `pre_backup` writes a marker, the orchestrator backs up,
|
||||
`pre_restore` destroys it, the orchestrator restores, `test_restore.py` asserts the marker is back.
|
||||
|
||||
### 5.3 Custom tier — `functional/` and `playwright/` ONLY
|
||||
### 5.3 Custom tier — canonical `custom/`
|
||||
|
||||
All custom-tier tests live under `tests/<recipe>/functional/` or `tests/<recipe>/playwright/`
|
||||
(discovery: `discovery.custom_tests`; the placement rule, §3). Run in the CUSTOM tier, after
|
||||
All custom-tier tests live under `tests/<recipe>/custom/` (discovery: `discovery.custom_tests`;
|
||||
the placement rule, §3). Deprecated `functional/` and `playwright/` dirs are still recognized
|
||||
with a warning during the migration window. Custom tests run in the CUSTOM tier, after
|
||||
restore, against the post-upgrade (PR-head) app. ALL discovered files run — cc-ci's and (if
|
||||
HC2-approved) repo-local's, additively.
|
||||
|
||||
Enrollment contract (`docs/enroll-recipe.md`): ≥2 NEW functional tests beyond ports of existing
|
||||
upstream checks; ported tests carry `SOURCE:` comments. Playwright tests get the shared
|
||||
Enrollment contract (`docs/enroll-recipe.md`): ≥2 NEW custom tests beyond ports of existing
|
||||
upstream checks; ported tests carry `SOURCE:` comments. Browser-driven custom tests get the shared
|
||||
browser/harness helpers (`harness.browser`); SSO recipes get `harness.sso`
|
||||
(`setup_keycloak_realm` — idempotent, `oidc_password_grant` — provider-pluggable). The documented
|
||||
import toolbox for custom tests is `from harness import lifecycle, sso, browser`.
|
||||
@ -268,7 +270,7 @@ deploy BASE (UPGRADE_BASE_VERSION or recipe_versions[-2]; EXTRA_ENV; install_ste
|
||||
→ BACKUP tier
|
||||
→ pre_restore(ctx) → restore
|
||||
→ RESTORE tier
|
||||
→ CUSTOM tier (functional/ + playwright/; deps via the `deps` fixture)
|
||||
→ CUSTOM tier (custom/; deps via the `deps` fixture)
|
||||
→ SCREENSHOT (best-effort, never affects the verdict)
|
||||
→ teardown (deps LAST)
|
||||
```
|
||||
@ -293,7 +295,7 @@ RECIPE=<recipe> PR=<n> REF=<sha> SRC=recipe-maintainers/<recipe> \
|
||||
meta (non-default): DEPLOY_TIMEOUT=1500 DEPS=['keycloak'] EXTRA_ENV='<hook>'
|
||||
hooks: ops.py[pre_backup,pre_upgrade](cc-ci) install_steps.sh(cc-ci) compose.ccci.yml(cc-ci)
|
||||
overlays: test_backup.py(cc-ci) test_restore.py(repo-local)
|
||||
custom tests: functional/=5 playwright/=2 (cc-ci)
|
||||
custom tests: custom/=7 (cc-ci)
|
||||
env overrides: (none)
|
||||
```
|
||||
|
||||
|
||||
@ -114,11 +114,12 @@ repo-local <recipe-repo>/tests/test_<op>.py (upstream-authoritative; gated
|
||||
Only ONE overlay source wins for a given op (repo-local > cc-ci); the generic floor runs **in
|
||||
addition** unless explicitly opted out.
|
||||
|
||||
**Custom (non-lifecycle) tests** — e.g. `functional/test_sso.py` — are **opt-in and additive**:
|
||||
**Custom (non-lifecycle) tests** — e.g. `custom/test_sso.py` — are **opt-in and additive**:
|
||||
they have no generic equivalent and run only when present, discovered from both locations
|
||||
(repo-local gated by the HC2 allowlist). Placement rule: custom tests live ONLY under
|
||||
`functional/` or `playwright/`; a top-level `test_*.py` is a lifecycle overlay and nothing else
|
||||
(top-level non-lifecycle files are not discovered).
|
||||
(repo-local gated by the HC2 allowlist). Placement rule: custom tests live under canonical
|
||||
`custom/`; deprecated `functional/` and `playwright/` aliases are still discovered with a loud
|
||||
warning so old recipe trees are not silently dropped. A top-level `test_*.py` is a lifecycle
|
||||
overlay and nothing else (top-level non-lifecycle files are not discovered).
|
||||
|
||||
### Pre-op seed hooks (per-recipe `ops.py`)
|
||||
|
||||
|
||||
Reference in New Issue
Block a user