feat(cfold): canonicalize custom test layout
continuous-integration/drone/push Build is failing

This commit is contained in:
autonomic-bot
2026-06-12 16:08:18 +00:00
parent 87928a9096
commit 44e02425ab
110 changed files with 306 additions and 241 deletions
+20 -21
View File
@@ -22,12 +22,11 @@ tests/<recipe>/
├── test_backup.py # optional backup overlay (runs ADDITIVELY alongside generic)
├── test_restore.py # optional restore overlay (runs ADDITIVELY alongside generic)
├── PARITY.md # Phase 2 P2: mapping table (recipe-maintainer tests → cc-ci tests)
── functional/ # Phase 2 P3: parity ports + ≥2 NEW recipe-specific tests
├── test_health_check.py # parity port of recipe-info/<recipe>/tests/health_check.py
├── test_<behavior>.py # ≥2 NEW recipe-specific functional tests
──
└── playwright/ # Phase 2 P6: browser flows where the app's core UX is a UI
└── test_<flow>.py
── custom/ # custom tier: parity ports + recipe-specific tests + browser flows
├── test_health_check.py # parity port of recipe-info/<recipe>/tests/health_check.py
├── test_<behavior>.py # ≥2 NEW recipe-specific tests
── test_<flow>.py # browser/UI flows where relevant
└── …
```
**A recipe is testable with ZERO config:** with no overlay files, the **generic lifecycle suite**
@@ -68,18 +67,18 @@ ops themselves are orchestrator-owned (you never call them from an overlay). The
Beyond the lifecycle overlays, each recipe carries (plan §4.1):
- **`PARITY.md`** — a mapping table from every `references/recipe-maintainer/recipe-info/<recipe>/
tests/*.py` to a comparable cc-ci test under `tests/<recipe>/functional/`, asserting the
tests/*.py` to a comparable cc-ci test under `tests/<recipe>/custom/`, asserting the
*same thing* (not a renamed file). A deliberate non-port is documented in `DECISIONS.md` with
a technical reason — never a silent omission.
- **`functional/`** — parity-port tests + **≥2 NEW recipe-specific functional tests** that
exercise the app's characteristic behavior (per plan §4.3 — e.g. "create-an-object +
read-it-back, and one more that touches a distinctive feature"). Each parity-port file carries
a `SOURCE = "recipe-info/<recipe>/tests/<file>"` comment near the top so audit is in-file.
- **`playwright/`** — browser flows where the recipe's core UX is a UI (P6).
- **`custom/`** — parity-port tests + **≥2 NEW recipe-specific tests** that exercise the app's
characteristic behavior (per plan §4.3 — e.g. "create-an-object + read-it-back, and one more
that touches a distinctive feature"). Browser/UI flows live in the same folder too. Each
parity-port file carries a `SOURCE = "recipe-info/<recipe>/tests/<file>"` comment near the top
so audit is in-file.
The orchestrator's **custom** tier discovers `test_*.py` in `tests/<recipe>/{functional,playwright}/`
ONLY (the placement rule, via `runner/harness/discovery.custom_tests` — a top-level `test_*.py`
is a lifecycle overlay and nothing else) and runs each as its own pytest against the same
The orchestrator's **custom** tier discovers `test_*.py` in canonical `tests/<recipe>/custom/`
(plus deprecated `functional/` / `playwright/` aliases during migration; discovery warns when it
uses them) and runs each as its own pytest against the same
`live_app` shared deployment. Lifecycle-named files (`test_install.py`/etc.) are **excluded**
from the custom tier even inside those subdirs (safety net against double-running).
@@ -176,7 +175,7 @@ shapes (proven on mumble, mailu, and the SSO-dependent suite):
**Non-HTTP protocol tests (mumble).** Reach a TCP service published `mode: host` (via a host-ports
overlay) at `127.0.0.1:<port>` — cc-ci runs tests on-host (cc-ci-run). mumble ships a stdlib protocol
client (`tests/mumble/functional/_mumble_proto.py`) doing the real TLS handshake → ServerSync; the
client (`tests/mumble/custom/_mumble_proto.py`) doing the real TLS handshake → ServerSync; the
recipe-specific tests assert channel presence and config round-trips (a deploy-set `WELCOME_TEXT`/
`USERS` value surfaces over the protocol — version-independent, non-vacuous).
@@ -244,7 +243,7 @@ tests/lasuite-docs/
├── test_backup.py # lifecycle backup overlay (marker captured)
├── test_restore.py # lifecycle restore overlay (marker restored to pre-mutation)
├── PARITY.md # parity-port mapping (P2)
└── functional/
└── custom/
├── test_health_check.py # parity port (SOURCE comment cites recipe-info file)
├── test_auth_required.py # specific: /api/v1.0/users/me/ → 401 without auth
└── test_oidc_with_keycloak.py # specific: full OIDC flow against the dep keycloak (uses
@@ -256,8 +255,8 @@ tests/lasuite-docs/
creds to `$CCCI_DEPS_FILE` — BEFORE the recipe deploy.
2. Deploy lasuite-docs (`lasu-<6hex>.ci.commoninternet.net`); `install_steps.sh` wires the OIDC
env into that one deploy.
3. Run install / upgrade / backup / restore + the 3 functional tests against the shared
deployment (custom tier).
3. Run install / upgrade / backup / restore + the 3 custom tests against the shared
deployment (custom tier).
4. Teardown lasuite-docs, then the keycloak dep (LAST), both with verify=True.
5. Print the run summary; non-zero exit code on any failure (DG4.1 deploy-count mismatch, tier
FAIL, dep teardown leak — all surfaced).
@@ -268,10 +267,10 @@ tests/lasuite-docs/
`COMPOSE_FILE=compose.yml:compose.mumbleweb.yml` for the base; `UPGRADE_EXTRA_ENV` adds the
native `compose.host-ports.yml` at PR-head so 64738 is host-published on latest; private
`_WELCOME_TEXT_MARKER`/`_MAX_USERS` constants; `READY_PROBE(ctx)` TCP 64738 — phase-aware via
the live COMPOSE_FILE), `functional/_mumble_proto.py` + the protocol/config-round-trip
the live COMPOSE_FILE), `custom/_mumble_proto.py` + the protocol/config-round-trip
tests, `ops.py`/`test_backup.py`/`test_restore.py` (sqlite P4). See §2.4.
- **Multi-service, dep-less, in-container functional — `tests/mailu/`**: `recipe_meta.py`
(`EXTRA_ENV(ctx)` with `TLS_FLAVOR=notls` + `MAIL_DOMAIN`/`HOSTNAMES`/`TRAEFIK_STACK_NAME`),
`functional/_mailu.py` (flask-CLI helpers), `test_mailbox.py` (create→config-export read-back),
`custom/_mailu.py` (flask-CLI helpers), `test_mailbox.py` (create→config-export read-back),
`test_mail_flow.py` (in-container sendmail→doveadm delivery). No backupbot → P4 N/A (PARITY.md +
DEFERRED.md). See §2.4.