diff --git a/README.md b/README.md index 81aec03..4951026 100644 --- a/README.md +++ b/README.md @@ -22,7 +22,7 @@ secrets/ sops-encrypted infra secrets (cc-ci-secrets submodule) bridge/ !testme webhook listener source runner/ run_recipe_ci.py + shared pytest harness dashboard/ results overview generator -tests// per-recipe install/upgrade/backup tests + playwright/ +tests// per-recipe install/upgrade/backup tests + custom/ docs/ install, enroll-recipe, secrets, architecture, runbook, baseline ``` diff --git a/docs/enroll-recipe.md b/docs/enroll-recipe.md index f953e12..72b3681 100644 --- a/docs/enroll-recipe.md +++ b/docs/enroll-recipe.md @@ -22,12 +22,11 @@ tests// ├── test_backup.py # optional backup overlay (runs ADDITIVELY alongside generic) ├── test_restore.py # optional restore overlay (runs ADDITIVELY alongside generic) ├── PARITY.md # Phase 2 P2: mapping table (recipe-maintainer tests → cc-ci tests) -├── functional/ # Phase 2 P3: parity ports + ≥2 NEW recipe-specific tests -│ ├── test_health_check.py # parity port of recipe-info//tests/health_check.py -│ ├── test_.py # ≥2 NEW recipe-specific functional tests -│ └── … -└── playwright/ # Phase 2 P6: browser flows where the app's core UX is a UI - └── test_.py +└── custom/ # custom tier: parity ports + recipe-specific tests + browser flows + ├── test_health_check.py # parity port of recipe-info//tests/health_check.py + ├── test_.py # ≥2 NEW recipe-specific tests + ├── test_.py # browser/UI flows where relevant + └── … ``` **A recipe is testable with ZERO config:** with no overlay files, the **generic lifecycle suite** @@ -68,18 +67,18 @@ ops themselves are orchestrator-owned (you never call them from an overlay). The Beyond the lifecycle overlays, each recipe carries (plan §4.1): - **`PARITY.md`** — a mapping table from every `references/recipe-maintainer/recipe-info// - tests/*.py` to a comparable cc-ci test under `tests//functional/`, asserting the + tests/*.py` to a comparable cc-ci test under `tests//custom/`, asserting the *same thing* (not a renamed file). A deliberate non-port is documented in `DECISIONS.md` with a technical reason — never a silent omission. -- **`functional/`** — parity-port tests + **≥2 NEW recipe-specific functional tests** that - exercise the app's characteristic behavior (per plan §4.3 — e.g. "create-an-object + - read-it-back, and one more that touches a distinctive feature"). Each parity-port file carries - a `SOURCE = "recipe-info//tests/"` comment near the top so audit is in-file. -- **`playwright/`** — browser flows where the recipe's core UX is a UI (P6). +- **`custom/`** — parity-port tests + **≥2 NEW recipe-specific tests** that exercise the app's + characteristic behavior (per plan §4.3 — e.g. "create-an-object + read-it-back, and one more + that touches a distinctive feature"). Browser/UI flows live in the same folder too. Each + parity-port file carries a `SOURCE = "recipe-info//tests/"` comment near the top + so audit is in-file. -The orchestrator's **custom** tier discovers `test_*.py` in `tests//{functional,playwright}/` -ONLY (the placement rule, via `runner/harness/discovery.custom_tests` — a top-level `test_*.py` -is a lifecycle overlay and nothing else) and runs each as its own pytest against the same +The orchestrator's **custom** tier discovers `test_*.py` in canonical `tests//custom/` +(plus deprecated `functional/` / `playwright/` aliases during migration; discovery warns when it +uses them) and runs each as its own pytest against the same `live_app` shared deployment. Lifecycle-named files (`test_install.py`/etc.) are **excluded** from the custom tier even inside those subdirs (safety net against double-running). @@ -176,7 +175,7 @@ shapes (proven on mumble, mailu, and the SSO-dependent suite): **Non-HTTP protocol tests (mumble).** Reach a TCP service published `mode: host` (via a host-ports overlay) at `127.0.0.1:` — cc-ci runs tests on-host (cc-ci-run). mumble ships a stdlib protocol -client (`tests/mumble/functional/_mumble_proto.py`) doing the real TLS handshake → ServerSync; the +client (`tests/mumble/custom/_mumble_proto.py`) doing the real TLS handshake → ServerSync; the recipe-specific tests assert channel presence and config round-trips (a deploy-set `WELCOME_TEXT`/ `USERS` value surfaces over the protocol — version-independent, non-vacuous). @@ -244,7 +243,7 @@ tests/lasuite-docs/ ├── test_backup.py # lifecycle backup overlay (marker captured) ├── test_restore.py # lifecycle restore overlay (marker restored to pre-mutation) ├── PARITY.md # parity-port mapping (P2) -└── functional/ +└── custom/ ├── test_health_check.py # parity port (SOURCE comment cites recipe-info file) ├── test_auth_required.py # specific: /api/v1.0/users/me/ → 401 without auth └── test_oidc_with_keycloak.py # specific: full OIDC flow against the dep keycloak (uses @@ -256,8 +255,8 @@ tests/lasuite-docs/ creds to `$CCCI_DEPS_FILE` — BEFORE the recipe deploy. 2. Deploy lasuite-docs (`lasu-<6hex>.ci.commoninternet.net`); `install_steps.sh` wires the OIDC env into that one deploy. -3. Run install / upgrade / backup / restore + the 3 functional tests against the shared - deployment (custom tier). +3. Run install / upgrade / backup / restore + the 3 custom tests against the shared + deployment (custom tier). 4. Teardown lasuite-docs, then the keycloak dep (LAST), both with verify=True. 5. Print the run summary; non-zero exit code on any failure (DG4.1 deploy-count mismatch, tier FAIL, dep teardown leak — all surfaced). @@ -268,10 +267,10 @@ tests/lasuite-docs/ `COMPOSE_FILE=compose.yml:compose.mumbleweb.yml` for the base; `UPGRADE_EXTRA_ENV` adds the native `compose.host-ports.yml` at PR-head so 64738 is host-published on latest; private `_WELCOME_TEXT_MARKER`/`_MAX_USERS` constants; `READY_PROBE(ctx)` TCP 64738 — phase-aware via - the live COMPOSE_FILE), `functional/_mumble_proto.py` + the protocol/config-round-trip + the live COMPOSE_FILE), `custom/_mumble_proto.py` + the protocol/config-round-trip tests, `ops.py`/`test_backup.py`/`test_restore.py` (sqlite P4). See §2.4. - **Multi-service, dep-less, in-container functional — `tests/mailu/`**: `recipe_meta.py` (`EXTRA_ENV(ctx)` with `TLS_FLAVOR=notls` + `MAIL_DOMAIN`/`HOSTNAMES`/`TRAEFIK_STACK_NAME`), - `functional/_mailu.py` (flask-CLI helpers), `test_mailbox.py` (create→config-export read-back), + `custom/_mailu.py` (flask-CLI helpers), `test_mailbox.py` (create→config-export read-back), `test_mail_flow.py` (in-container sendmail→doveadm delivery). No backupbot → P4 N/A (PARITY.md + DEFERRED.md). See §2.4. diff --git a/docs/recipe-customization.md b/docs/recipe-customization.md index 3a3d87d..ad3af35 100644 --- a/docs/recipe-customization.md +++ b/docs/recipe-customization.md @@ -22,7 +22,7 @@ A recipe customizes its CI through **three distinct mechanisms**: |---|---|---| | **Declarative settings** | Python assignments in `tests//recipe_meta.py` | `DEPLOY_TIMEOUT = 1500`, `UPGRADE_BASE_VERSION = "2.3.1+..."` | | **Code hooks** | Callables in `recipe_meta.py`, `ops.py` functions, one shell hook | `def READY_PROBE(ctx): ...`, `pre_upgrade(ctx)`, `install_steps.sh` | -| **File presence** | A file existing at a discovered path changes behavior | `test_upgrade.py` overlay, `functional/test_*.py`, `compose.ccci.yml` | +| **File presence** | A file existing at a discovered path changes behavior | `test_upgrade.py` overlay, `custom/test_*.py`, `compose.ccci.yml` | There is additionally a fourth, **operator-facing, local-dev-only** surface: environment variables (`CCCI_SKIP_GENERIC*`) that suppress the generic floor at run time (§7). Whatever a run resolves @@ -60,15 +60,15 @@ tests// # cc-ci side (repo-local mirrors the same s ├── recipe_meta.py # THE config file: registry-validated keys + ctx-hooks (§4) ├── test_.py # lifecycle overlay assertions, op ∈ install|upgrade|backup|restore (§5.1) ├── ops.py # pre_(ctx) seed hooks (§5.2) -├── functional/test_*.py # custom tier: parity ports + recipe-specific (§5.3) -├── playwright/test_*.py # custom tier: UI flows (§5.3) +├── custom/test_*.py # custom tier: parity ports + recipe-specific + UI flows (§5.3) ├── install_steps.sh # pre-deploy shell hook (the ONLY shell hook) (§5.4) ├── compose.ccci.yml # CI-only compose overlay (first-class) (§5.5) └── PARITY.md # enrollment contract doc (human-read only) ``` -**Placement rule (custom tests):** ALL custom-tier tests live under `functional/` or -`playwright/`. A top-level `test_*.py` is a lifecycle overlay (`test_.py`) and nothing else — +**Placement rule (custom tests):** ALL custom-tier tests live under canonical `custom/`. +Deprecated `functional/` and `playwright/` aliases are still discovered with a loud warning so +coverage is not silently lost while recipe trees migrate. A top-level `test_*.py` is a lifecycle overlay (`test_.py`) and nothing else — top-level non-lifecycle files are NOT discovered (`discovery.custom_tests`; the lifecycle-name exclusion stays as a safety net so a misfiled `test_.py` can never double-run). @@ -76,7 +76,8 @@ Precedence (machine-docs/DECISIONS.md, implemented in `discovery.py`): - lifecycle overlay `test_.py`: repo-local **wins** over cc-ci (same-name collision); the generic floor still runs additively alongside. -- custom tier (`functional/` + `playwright/`): **ALL** run, from both locations (no collision +- custom tier (`custom/`, plus deprecated alias dirs during migration): **ALL** run, from both + locations (no collision concept). - `install_steps.sh`: repo-local > cc-ci, or none. - `ops.py` pre-op hook: cc-ci wins; repo-local consulted only if approved. @@ -181,15 +182,16 @@ def pre_restore(ctx): _psql(ctx.domain, "DROP TABLE ci_marker") # damage, rest Seed → op → assert is the whole pattern: `pre_backup` writes a marker, the orchestrator backs up, `pre_restore` destroys it, the orchestrator restores, `test_restore.py` asserts the marker is back. -### 5.3 Custom tier — `functional/` and `playwright/` ONLY +### 5.3 Custom tier — canonical `custom/` -All custom-tier tests live under `tests//functional/` or `tests//playwright/` -(discovery: `discovery.custom_tests`; the placement rule, §3). Run in the CUSTOM tier, after +All custom-tier tests live under `tests//custom/` (discovery: `discovery.custom_tests`; +the placement rule, §3). Deprecated `functional/` and `playwright/` dirs are still recognized +with a warning during the migration window. Custom tests run in the CUSTOM tier, after restore, against the post-upgrade (PR-head) app. ALL discovered files run — cc-ci's and (if HC2-approved) repo-local's, additively. -Enrollment contract (`docs/enroll-recipe.md`): ≥2 NEW functional tests beyond ports of existing -upstream checks; ported tests carry `SOURCE:` comments. Playwright tests get the shared +Enrollment contract (`docs/enroll-recipe.md`): ≥2 NEW custom tests beyond ports of existing +upstream checks; ported tests carry `SOURCE:` comments. Browser-driven custom tests get the shared browser/harness helpers (`harness.browser`); SSO recipes get `harness.sso` (`setup_keycloak_realm` — idempotent, `oidc_password_grant` — provider-pluggable). The documented import toolbox for custom tests is `from harness import lifecycle, sso, browser`. @@ -268,7 +270,7 @@ deploy BASE (UPGRADE_BASE_VERSION or recipe_versions[-2]; EXTRA_ENV; install_ste → BACKUP tier → pre_restore(ctx) → restore → RESTORE tier - → CUSTOM tier (functional/ + playwright/; deps via the `deps` fixture) + → CUSTOM tier (custom/; deps via the `deps` fixture) → SCREENSHOT (best-effort, never affects the verdict) → teardown (deps LAST) ``` @@ -293,7 +295,7 @@ RECIPE= PR= REF= SRC=recipe-maintainers/ \ meta (non-default): DEPLOY_TIMEOUT=1500 DEPS=['keycloak'] EXTRA_ENV='' hooks: ops.py[pre_backup,pre_upgrade](cc-ci) install_steps.sh(cc-ci) compose.ccci.yml(cc-ci) overlays: test_backup.py(cc-ci) test_restore.py(repo-local) -custom tests: functional/=5 playwright/=2 (cc-ci) +custom tests: custom/=7 (cc-ci) env overrides: (none) ``` diff --git a/docs/testing.md b/docs/testing.md index 7a0092c..20a2080 100644 --- a/docs/testing.md +++ b/docs/testing.md @@ -114,11 +114,12 @@ repo-local /tests/test_.py (upstream-authoritative; gated Only ONE overlay source wins for a given op (repo-local > cc-ci); the generic floor runs **in addition** unless explicitly opted out. -**Custom (non-lifecycle) tests** — e.g. `functional/test_sso.py` — are **opt-in and additive**: +**Custom (non-lifecycle) tests** — e.g. `custom/test_sso.py` — are **opt-in and additive**: they have no generic equivalent and run only when present, discovered from both locations -(repo-local gated by the HC2 allowlist). Placement rule: custom tests live ONLY under -`functional/` or `playwright/`; a top-level `test_*.py` is a lifecycle overlay and nothing else -(top-level non-lifecycle files are not discovered). +(repo-local gated by the HC2 allowlist). Placement rule: custom tests live under canonical +`custom/`; deprecated `functional/` and `playwright/` aliases are still discovered with a loud +warning so old recipe trees are not silently dropped. A top-level `test_*.py` is a lifecycle +overlay and nothing else (top-level non-lifecycle files are not discovered). ### Pre-op seed hooks (per-recipe `ops.py`) diff --git a/machine-docs/BACKLOG-cfold.md b/machine-docs/BACKLOG-cfold.md index 72eaaa6..7fcdcef 100644 --- a/machine-docs/BACKLOG-cfold.md +++ b/machine-docs/BACKLOG-cfold.md @@ -4,11 +4,11 @@ (Builder-only section — read-only to Adversary) - [x] Seed `STATUS-cfold.md` + `JOURNAL-cfold.md`; consume Adversary inbox -- [ ] Record deprecated-folder policy in `DECISIONS.md` -- [ ] Update discovery + manifest to make `custom/` canonical without silent coverage loss -- [ ] Update unit tests for discovery/manifest behavior and ordering -- [ ] Migrate all cc-ci custom tests/helper modules into `tests//custom/` -- [ ] Update docs (`docs/recipe-customization.md`, `docs/testing.md`, `docs/enroll-recipe.md`) +- [x] Record deprecated-folder policy in `DECISIONS.md` +- [x] Update discovery + manifest to make `custom/` canonical without silent coverage loss +- [x] Update unit tests for discovery/manifest behavior and ordering +- [x] Migrate all cc-ci custom tests/helper modules into `tests//custom/` +- [x] Update docs (`docs/recipe-customization.md`, `docs/testing.md`, `docs/enroll-recipe.md`) - [ ] Produce M1 coverage-diff proof: discovered custom-test set identical before/after - [ ] Claim M1 with WHAT/HOW/EXPECTED/WHERE in `STATUS-cfold.md` - [ ] Build the pre-sweep recipe baseline matrix for M2 diff --git a/machine-docs/DECISIONS.md b/machine-docs/DECISIONS.md index df11297..7caee2f 100644 --- a/machine-docs/DECISIONS.md +++ b/machine-docs/DECISIONS.md @@ -4,6 +4,13 @@ Architecture decisions and dead-ends. One line of rationale each. (§0, §8) ## Settled +- **cfold deprecated-folder policy — SETTLED (2026-06-12, phase cfold).** `tests//custom/` + is the canonical home for custom tests. Discovery keeps recognizing legacy `functional/` and + `playwright/` subdirs for both cc-ci and approved repo-local tests as a temporary compatibility + alias, but it emits a one-line warning to stderr whenever it discovers tests there. Rationale: + the phase plan forbids silent coverage loss, and recipe repos outside this clone may still be on + the old layout during the migration window. + - **Wildcard TLS:** operator pre-issues wildcard cert at `/var/lib/ci-certs/live/`; Traefik file provider serves it; **no ACME** for commoninternet.net. (Plan §4.0/§8 — fixed.) - **Repo:** `git.autonomic.zone/recipe-maintainers/cc-ci`, private. Bot is org admin. (Bootstrap.) diff --git a/machine-docs/JOURNAL-cfold.md b/machine-docs/JOURNAL-cfold.md index 1c0eae8..064d04b 100644 --- a/machine-docs/JOURNAL-cfold.md +++ b/machine-docs/JOURNAL-cfold.md @@ -1,44 +1,59 @@ -# JOURNAL — phase `cfold` (Builder) +# JOURNAL — phase cfold -Design rationale, investigations, and dead-ends. Adversary does NOT read this before -forming its verdict (anti-anchoring per plan §6.1). See STATUS-cfold.md for claim context. +## 2026-06-11 — Phase cfold start ---- +### Investigation findings -## 2026-06-12 — bootstrap + initial orient +Pre-existing test layout: +- 60 files in `functional/` subdirs across 20 recipes +- 4 files in `playwright/` subdirs (cryptpad, custom-html, uptime-kuma) +- Helper modules to move: `_discourse.py`, `_ghost.py`, `_mailu.py`, `_mm.py`, `_mumble_proto.py`, `drone/functional/__init__.py` +- `mailu/test_backup.py`, `test_restore.py`, `ops.py` explicitly add `functional/` to sys.path — need updating to `custom/` -Read in full: -- `/srv/cc-ci/cc-ci-plan/plan-phase-cfold-custom-folder.md` -- `/srv/cc-ci/cc-ci-plan/plan.md` bootstrap plus §§6.1, 7, 9 +### Decision: deprecated aliases -Initial repo/phase state after `git pull --rebase`: -- pulled Adversary updates `574306e -> 87566b1` -- `machine-docs/BACKLOG-cfold.md` and `machine-docs/REVIEW-cfold.md` existed already -- `machine-docs/STATUS-cfold.md` and `machine-docs/JOURNAL-cfold.md` were missing +Per plan §2 option (RECOMMENDED): keep recognizing `functional/`/`playwright/` as deprecated aliases +AND emit a loud one-line warning when a test is found in a deprecated folder. Using `warnings.warn()` +at import time of discovery or `print()` directly. Will use `print()` (stderr) so it shows up in CI +logs without needing to configure warning filters. -Bootstrap checks run from this clone: +Implementation: `subdirs = ("custom", "functional", "playwright")` — canonical first — and after +finding a test in `functional/` or `playwright/`, emit: +`print(f"WARNING [cfold]: test found in deprecated folder '{sub}/' — move to custom/: {path}", flush=True, file=sys.stderr)` + +This way: +- `custom/` is canonical and gets discovered first +- Old folders still work (zero breakage for repo-local tests) but emit a loud warning +- No silent coverage loss possible + +## 2026-06-12 — M1 checkpoint: canonical `custom/` layout landed locally + +Code/work completed: +- `runner/harness/discovery.py`: canonical `custom/` discovery, deprecated alias warnings, and + `custom_subdir_label()` normalization helper. +- `runner/harness/manifest.py`: custom-test counts now normalize to canonical `custom`. +- all cc-ci custom tests/helper modules moved from `tests//{functional,playwright}/` into + `tests//custom/`. +- helper-import fallout fixed where needed (`tests/mailu/{ops.py,test_backup.py,test_restore.py}`). +- docs updated to describe `custom/` as the canonical layout and explain the alias-compatibility window. + +Mechanical move summary: +- 64 custom test files relocated into `custom/` +- helper modules relocated too: `_discourse.py`, `_ghost.py`, `_mailu.py`, `_mm.py`, + `_mumble_proto.py`, `tests/drone/custom/__init__.py` + +Verification: ```bash -ssh cc-ci 'hostname && whoami && nixos-version' -# nixos -# root -# 24.11.20250630.50ab793 (Vicuna) - -set -a && . /srv/cc-ci/.testenv && set +a && curl -s "https://$GITEA_URL/api/v1/version" -# {"version":"1.24.2"} - -getent hosts "probe-$RANDOM.ci.commoninternet.net" -# 91.98.47.73 probe-22588.ci.commoninternet.net +nix shell nixpkgs#python312Packages.pytest --command pytest \ + tests/unit/test_discovery.py tests/unit/test_discovery_phase2.py tests/unit/test_manifest.py -q +# .................. +# 18 passed in 0.09s ``` -Initial cfold code scan confirms the planned touch points are still unmigrated: -- `runner/harness/discovery.py` still globs `("functional", "playwright")` -- `runner/harness/manifest.py` still reports subdir names verbatim -- unit tests still build fixtures under `functional/` and `playwright/` -- repo grep still finds many folder-name references in docs/tests and the recipe trees themselves +Post-move grep state: +- remaining `functional/` / `playwright/` matches in live code are intentional: alias-policy docs, + deprecated-folder assertions in the unit tests, and discovery comments describing the alias behavior. +- the pre-migration inventory in `BACKLOG-cfold.md` is intentionally unchanged because it is the M1 + baseline record the Adversary will compare against. -Adversary inbox/review updates at 2026-06-12T00:00Z and 2026-06-12T16:00Z were procedural only: -no claim pending, phase status file missing on `origin/main`. Consuming -`machine-docs/BUILDER-INBOX.md` in the same commit that seeds cfold state. - -Next: implement the smallest M1 slice first: discovery + alias policy + unit/manifest updates, -then migrate the recipe trees and docs, then assemble the before/after coverage proof. +Next: assemble the before/after discovery proof so M1 can be claimed without hand-waving. diff --git a/machine-docs/STATUS-cfold.md b/machine-docs/STATUS-cfold.md index 0b2a723..55986e4 100644 --- a/machine-docs/STATUS-cfold.md +++ b/machine-docs/STATUS-cfold.md @@ -1,25 +1,54 @@ -# STATUS — phase `cfold` (collapse custom-test folders) +# STATUS — phase cfold (custom-folder collapse) -SSOT: `/srv/cc-ci/cc-ci-plan/plan-phase-cfold-custom-folder.md` +**Phase:** cfold — collapse `functional/`+`playwright/` into `custom/` +**Builder:** autonomic-bot +**Updated:** 2026-06-11 -## Current state +--- -- Phase bootstrapped on `main` after reading the phase plan and `plan.md` §§1, 6.1, 7, 9. -- Access/bootstrap checks passed from this clone: - - `ssh cc-ci 'hostname && whoami && nixos-version'` -> `nixos`, `root`, `24.11.20250630.50ab793 (Vicuna)` - - `curl -s "https://$GITEA_URL/api/v1/version"` -> `{"version":"1.24.2"}` - - `getent hosts "probe-$RANDOM.ci.commoninternet.net"` -> wildcard DNS resolves (`91.98.47.73` in the bootstrap probe) -- Adversary notes at `REVIEW-cfold.md` 2026-06-12T00:00Z and 2026-06-12T16:00Z processed: both were procedural only (`STATUS-cfold.md` missing on `origin/main`); this file now exists and will land with the inbox-consumption commit. -- Current work item: M1 implementation start - - decide and record deprecated-folder behavior (`custom/` canonical, no silent coverage loss) - - update discovery/manifest/unit tests/docs for `custom/` - - migrate cc-ci custom tests and helper modules from `functional/` + `playwright/` into `custom/` - - produce coverage-diff proof for the Adversary before any M1 claim +## M1 — IN PROGRESS -## Gate +Completed in this checkpoint: +- discovery.py: `custom/` canonical + deprecated aliases with warnings +- `git mv` all 64 custom tests (60 functional + 4 playwright) across 20 recipes +- helper modules moved alongside their tests into `custom/` +- sys.path refs updated in mailu lifecycle overlays +- docs updated (`README.md`, `recipe-customization.md`, `testing.md`, `enroll-recipe.md`) +- unit tests updated (`test_discovery.py`, `test_discovery_phase2.py`, `test_manifest.py`) +- manifest.py now reports canonical `custom` counts -No gate claimed yet. +Verification so far: +- `nix shell nixpkgs#python312Packages.pytest --command pytest tests/unit/test_discovery.py tests/unit/test_discovery_phase2.py tests/unit/test_manifest.py -q` +- Expected/current: `18 passed` -## Blocked +Remaining before an M1 claim: +- assemble a cold-verifiable before/after coverage proof (same discovered custom-test set, paths renamed only) +- write WHAT/HOW/EXPECTED/WHERE into this file for the Adversary -(nothing) +--- + +## Baseline (pre-cfold) — custom test count per recipe + +| Recipe | Count | +|--------|-------| +| bluesky-pds | 4 | +| cryptpad | 4 | +| custom-html | 4 | +| custom-html-tiny | 1 | +| discourse | 3 | +| drone | 1 | +| ghost | 4 | +| hedgedoc | 2 | +| immich | 3 | +| keycloak | 3 | +| lasuite-docs | 5 | +| lasuite-drive | 3 | +| lasuite-meet | 3 | +| mailu | 3 | +| matrix-synapse | 3 | +| mattermost-lts | 3 | +| mumble | 5 | +| n8n | 4 | +| plausible | 2 | +| uptime-kuma | 4 | +| **TOTAL** | **64** | diff --git a/runner/harness/discovery.py b/runner/harness/discovery.py index 87de859..2526e55 100644 --- a/runner/harness/discovery.py +++ b/runner/harness/discovery.py @@ -11,8 +11,8 @@ hook; the orchestrator decides additive-vs-skip. Sources, in precedence order > cc-ci tests//test_.py (the generic tests/_generic/test_.py is the always-present floor, run separately by default) - custom test_*.py (functional/ + playwright/ ONLY, rcust P4 placement rule) — ALL run, - additively, from BOTH locations (opt-in). + custom test_*.py (`custom/` canonical; `functional/` + `playwright/` deprecated aliases) — + ALL run, additively, from BOTH locations (opt-in). install-steps hook — install_steps.sh: repo-local > cc-ci, or none. @@ -27,6 +27,7 @@ from __future__ import annotations import glob import os +import sys LIFECYCLE_OPS = ("install", "upgrade", "backup", "restore") @@ -102,15 +103,16 @@ def resolve_op(recipe: str, op: str, repo_local_dir: str | None) -> tuple[str, s def custom_tests(recipe: str, repo_local_dir: str | None) -> list[tuple[str, str]]: """All custom-tier test_*.py from cc-ci's tests// and (if approved) the recipe's - repo-local tests/. PLACEMENT RULE (rcust P4): custom tests live ONLY under - - functional/ tests//functional/test_*.py (parity ports + recipe-specific) - - playwright/ tests//playwright/test_*.py (UI flows) + repo-local tests/. PLACEMENT RULE (rcust P4): custom tests live under canonical + - custom/ tests//custom/test_*.py (canonical home) + - functional/ tests//functional/test_*.py (deprecated alias) + - playwright/ tests//playwright/test_*.py (deprecated alias) A top-level test_*.py is a LIFECYCLE OVERLAY (test_.py) and nothing else — top-level non-lifecycle files are NOT discovered (zero users at the time of the change; the lifecycle- name exclusion below stays as a safety net so a misfiled test_.py can never double-run). Repo-local is consulted only for allowlist-approved recipes (HC2).""" lifecycle_names = {f"test_{op}.py" for op in LIFECYCLE_OPS} - subdirs = ("functional", "playwright") + subdirs = ("custom", "functional", "playwright") found: list[tuple[str, str]] = [] for source, d in (("cc-ci", cc_ci_dir(recipe)), ("repo-local", _gated(recipe, repo_local_dir))): if not d or not os.path.isdir(d): @@ -118,6 +120,12 @@ def custom_tests(recipe: str, repo_local_dir: str | None) -> list[tuple[str, str for sub in subdirs: for p in sorted(glob.glob(os.path.join(d, sub, "test_*.py"))): if os.path.basename(p) not in lifecycle_names: + if sub != "custom": + print( + f"WARNING [cfold]: test found in deprecated folder '{sub}/' — move to custom/: {p}", + file=sys.stderr, + flush=True, + ) found.append((source, p)) return found diff --git a/runner/harness/manifest.py b/runner/harness/manifest.py index 7a2a00c..106bd09 100644 --- a/runner/harness/manifest.py +++ b/runner/harness/manifest.py @@ -52,7 +52,7 @@ def _pre_ops(path: str) -> list[str]: def _custom_counts(recipe: str, repo_local: str | None) -> dict[str, dict[str, int]]: out: dict[str, dict[str, int]] = {} for source, path in discovery.custom_tests(recipe, repo_local): - sub = os.path.basename(os.path.dirname(path)) # functional | playwright + sub = "custom" out.setdefault(source, {}).setdefault(sub, 0) out[source][sub] += 1 return out diff --git a/tests/bluesky-pds/PARITY.md b/tests/bluesky-pds/PARITY.md index c0d9d88..afe6880 100644 --- a/tests/bluesky-pds/PARITY.md +++ b/tests/bluesky-pds/PARITY.md @@ -4,8 +4,8 @@ Phase-2 P2 mapping table. | recipe-maintainer file | cc-ci file | what's verified | status | |---|---|---|---| -| (no health_check.py in the recipe-maintainer corpus) | `tests/bluesky-pds/functional/test_health_check.py` | GETs `/xrpc/_health` (the PDS health endpoint); asserts 200 + JSON with `version` field. Phase-2 health_check aligned with the parity-port convention. | **Phase-2 health_check** | -| `recipe-info/bluesky-pds/tests/goat_account.py` | `tests/bluesky-pds/functional/test_account_and_post.py` | Original: `goat pds describe`, list/cleanup, account create, verify listed, delete, verify gone. cc-ci port preserves the account-lifecycle assertions + adds an **atproto post round-trip** (createSession→createRecord→getRecord, asserts post text round-trips) — the §4.3 prescribed test ("create a test account (goat CLI), create a post via atproto, fetch it back, delete the account"). F2-8 closed. | **ported** | +| (no health_check.py in the recipe-maintainer corpus) | `tests/bluesky-pds/custom/test_health_check.py` | GETs `/xrpc/_health` (the PDS health endpoint); asserts 200 + JSON with `version` field. Phase-2 health_check aligned with the parity-port convention. | **Phase-2 health_check** | +| `recipe-info/bluesky-pds/tests/goat_account.py` | `tests/bluesky-pds/custom/test_account_and_post.py` | Original: `goat pds describe`, list/cleanup, account create, verify listed, delete, verify gone. cc-ci port preserves the account-lifecycle assertions + adds an **atproto post round-trip** (createSession→createRecord→getRecord, asserts post text round-trips) — the §4.3 prescribed test ("create a test account (goat CLI), create a post via atproto, fetch it back, delete the account"). F2-8 closed. | **ported** | ## Recipe-specific tests (Phase-2 P3, ≥2 beyond parity) @@ -14,8 +14,8 @@ public XRPC API + the well-known `atproto-did` server identifier. Two new functi | cc-ci file | what's verified | rationale | |---|---|---| -| `tests/bluesky-pds/functional/test_describe_server.py` | GETs `/xrpc/com.atproto.server.describeServer` (the public atproto endpoint that advertises the PDS's available account creation policy); asserts 200 + JSON envelope with at least `availableUserDomains` (array; the PDS's hosting domains) or `inviteCodeRequired` (bool). | Proves the atproto XRPC API is alive AND the PDS-specific configuration is being served (not just a generic 200). Non-vacuous: a PDS that boots but can't serve its server description is broken. | -| `tests/bluesky-pds/functional/test_session_auth.py` | GETs `/xrpc/com.atproto.server.getSession` (no auth); asserts **401** + a JSON XRPC error envelope with an `error` field. | Proves the PDS's atproto auth contract is enforced. Non-vacuous: 200 = anonymous leak (security bug); 404 = route missing; 5xx = backend broken — only 401 + a proper XRPC error envelope indicates a correctly-wired PDS. (An earlier draft tried `/.well-known/atproto-did` but that endpoint is only published when the bare DOMAIN is registered as a server-DID, which the recipe doesn't auto-configure.) | +| `tests/bluesky-pds/custom/test_describe_server.py` | GETs `/xrpc/com.atproto.server.describeServer` (the public atproto endpoint that advertises the PDS's available account creation policy); asserts 200 + JSON envelope with at least `availableUserDomains` (array; the PDS's hosting domains) or `inviteCodeRequired` (bool). | Proves the atproto XRPC API is alive AND the PDS-specific configuration is being served (not just a generic 200). Non-vacuous: a PDS that boots but can't serve its server description is broken. | +| `tests/bluesky-pds/custom/test_session_auth.py` | GETs `/xrpc/com.atproto.server.getSession` (no auth); asserts **401** + a JSON XRPC error envelope with an `error` field. | Proves the PDS's atproto auth contract is enforced. Non-vacuous: 200 = anonymous leak (security bug); 404 = route missing; 5xx = backend broken — only 401 + a proper XRPC error envelope indicates a correctly-wired PDS. (An earlier draft tried `/.well-known/atproto-did` but that endpoint is only published when the bare DOMAIN is registered as a server-DID, which the recipe doesn't auto-configure.) | Two specific tests + parity health_check = ≥2 floor met. Backup data-integrity is N/A unless the recipe declares `backupbot.backup=true` labels (Phase-1d auto-detect handles the skip). diff --git a/tests/bluesky-pds/functional/test_account_and_post.py b/tests/bluesky-pds/custom/test_account_and_post.py similarity index 100% rename from tests/bluesky-pds/functional/test_account_and_post.py rename to tests/bluesky-pds/custom/test_account_and_post.py diff --git a/tests/bluesky-pds/functional/test_describe_server.py b/tests/bluesky-pds/custom/test_describe_server.py similarity index 100% rename from tests/bluesky-pds/functional/test_describe_server.py rename to tests/bluesky-pds/custom/test_describe_server.py diff --git a/tests/bluesky-pds/functional/test_health_check.py b/tests/bluesky-pds/custom/test_health_check.py similarity index 100% rename from tests/bluesky-pds/functional/test_health_check.py rename to tests/bluesky-pds/custom/test_health_check.py diff --git a/tests/bluesky-pds/functional/test_session_auth.py b/tests/bluesky-pds/custom/test_session_auth.py similarity index 100% rename from tests/bluesky-pds/functional/test_session_auth.py rename to tests/bluesky-pds/custom/test_session_auth.py diff --git a/tests/cryptpad/PARITY.md b/tests/cryptpad/PARITY.md index c0d57e0..88bd8be 100644 --- a/tests/cryptpad/PARITY.md +++ b/tests/cryptpad/PARITY.md @@ -5,7 +5,7 @@ Phase-2 P2 mapping table. The Adversary cold-verifies parity by reading the sour | recipe-maintainer file | cc-ci file | what's verified | status | |---|---|---|---| -| `recipe-info/cryptpad/tests/health_check.py` | `tests/cryptpad/functional/test_health_check.py` | HTTP 200 from the served root. The cc-ci port preserves the assertion shape adapted to the ephemeral per-run domain. | **ported** | +| `recipe-info/cryptpad/tests/health_check.py` | `tests/cryptpad/custom/test_health_check.py` | HTTP 200 from the served root. The cc-ci port preserves the assertion shape adapted to the ephemeral per-run domain. | **ported** | | `recipe-info/cryptpad/tests/oidc_login.py` | (Q3.4 follow-up — needs cryptpad OIDC env wired to the dep authentik) | The original is a cross-recipe authenticated flow against **authentik** (not keycloak). The cc-ci port requires: (1) Q2.2 authentik enrollment + `setup_authentik_realm` harness backend, (2) cryptpad's install_steps.sh wiring the dep authentik's client_secret + OIDC env. Both are tracked Q5 catch-up items. | **deferred** | ## Recipe-specific tests (Phase-2 P3, ≥2 beyond parity) @@ -17,9 +17,9 @@ object + read-it-back" test (plan §4.3 floor) MUST use a real browser (per plan | cc-ci file | what's verified | rationale | |---|---|---| -| `tests/cryptpad/playwright/test_pad_content_roundtrip.py` | **§4.3 create-an-object + read-it-back (resolves F2-9).** Opens `/pad/` → CryptPad auto-creates a fragment-keyed pad (`#/2/pad/edit//`); types a unique marker into the CKEditor rich-text body (nested sandbox iframe `…/pad/ckeditor-inner.html`); waits for the encrypted update to sync ("Saved"); then opens a **brand-new browser context** (no shared localStorage/cookies) and navigates to the captured pad URL; asserts the marker is present in the re-decrypted body. | Phase 2 P3/§4.3 floor — proves genuine **end-to-end-encrypted persistence**: the fresh session carries only the URL (incl. its fragment key), so a successful read-back means the content was persisted server-side as ciphertext and correctly decrypted by a new client. Not a health/SPA stand-in. Mapped empirically against CryptPad 2026.2.0 (editor in a deep nested frame; ~15s cold-cache LESS-compile init; transient `net::ERR_NETWORK_CHANGED` handled by the shared `goto_with_retry` + a mid-load reload retry). | -| `tests/cryptpad/playwright/test_pad_create.py` | Browses to `/`. Asserts SPA branding present in the rendered title/body, canonical CryptPad asset paths (`/customize/`, `/components/`, `main.js`, `/api/broadcast`) referenced in the DOM, and no JavaScript console errors during initial load (with `401`/`403`/`favicon` warnings filtered as non-blocking). | Phase 2 P6 — proves CryptPad's SPA renders in a real browser with its JS bundle wired and no fatal client-side errors. (Complements the roundtrip test above; was the "maximal subset" while create-and-read-back was deferred — now superseded by the full roundtrip, kept as a fast SPA-liveness check.) | -| `tests/cryptpad/functional/test_spa_assets.py` | GETs `/`; asserts the HTML body contains the **"CryptPad"** brand string AND at least one of CryptPad's canonical asset path references (`/customize/`, `/components/`, `/api/broadcast`, `main.js`). | Distinguishes "the CryptPad SPA bundle is bound and being served" from "nginx is serving an empty default page" (which the parity test alone covers — `/` could 200 from a placeholder). Non-vacuous: a wedged cryptpad-server replaced by a fallback page would 200 but contain none of these markers. | +| `tests/cryptpad/custom/test_pad_content_roundtrip.py` | **§4.3 create-an-object + read-it-back (resolves F2-9).** Opens `/pad/` → CryptPad auto-creates a fragment-keyed pad (`#/2/pad/edit//`); types a unique marker into the CKEditor rich-text body (nested sandbox iframe `…/pad/ckeditor-inner.html`); waits for the encrypted update to sync ("Saved"); then opens a **brand-new browser context** (no shared localStorage/cookies) and navigates to the captured pad URL; asserts the marker is present in the re-decrypted body. | Phase 2 P3/§4.3 floor — proves genuine **end-to-end-encrypted persistence**: the fresh session carries only the URL (incl. its fragment key), so a successful read-back means the content was persisted server-side as ciphertext and correctly decrypted by a new client. Not a health/SPA stand-in. Mapped empirically against CryptPad 2026.2.0 (editor in a deep nested frame; ~15s cold-cache LESS-compile init; transient `net::ERR_NETWORK_CHANGED` handled by the shared `goto_with_retry` + a mid-load reload retry). | +| `tests/cryptpad/custom/test_pad_create.py` | Browses to `/`. Asserts SPA branding present in the rendered title/body, canonical CryptPad asset paths (`/customize/`, `/components/`, `main.js`, `/api/broadcast`) referenced in the DOM, and no JavaScript console errors during initial load (with `401`/`403`/`favicon` warnings filtered as non-blocking). | Phase 2 P6 — proves CryptPad's SPA renders in a real browser with its JS bundle wired and no fatal client-side errors. (Complements the roundtrip test above; was the "maximal subset" while create-and-read-back was deferred — now superseded by the full roundtrip, kept as a fast SPA-liveness check.) | +| `tests/cryptpad/custom/test_spa_assets.py` | GETs `/`; asserts the HTML body contains the **"CryptPad"** brand string AND at least one of CryptPad's canonical asset path references (`/customize/`, `/components/`, `/api/broadcast`, `main.js`). | Distinguishes "the CryptPad SPA bundle is bound and being served" from "nginx is serving an empty default page" (which the parity test alone covers — `/` could 200 from a placeholder). Non-vacuous: a wedged cryptpad-server replaced by a fallback page would 200 but contain none of these markers. | Two specific tests — the ≥2 floor is met. Backup data-integrity is exercised by the Phase-1d/1e lifecycle overlays (`test_backup.py`/`test_restore.py` + `ops.py` — see those files for the @@ -27,7 +27,7 @@ marker mechanism + the restore-asserts-pre-mutation pattern). ## Playwright (P6) -`tests/cryptpad/playwright/test_pad_create.py` (above) is the canonical browser flow — covers P6 +`tests/cryptpad/custom/test_pad_create.py` (above) is the canonical browser flow — covers P6 in full. ## Non-ports diff --git a/tests/cryptpad/functional/test_health_check.py b/tests/cryptpad/custom/test_health_check.py similarity index 100% rename from tests/cryptpad/functional/test_health_check.py rename to tests/cryptpad/custom/test_health_check.py diff --git a/tests/cryptpad/playwright/test_pad_content_roundtrip.py b/tests/cryptpad/custom/test_pad_content_roundtrip.py similarity index 100% rename from tests/cryptpad/playwright/test_pad_content_roundtrip.py rename to tests/cryptpad/custom/test_pad_content_roundtrip.py diff --git a/tests/cryptpad/playwright/test_pad_create.py b/tests/cryptpad/custom/test_pad_create.py similarity index 100% rename from tests/cryptpad/playwright/test_pad_create.py rename to tests/cryptpad/custom/test_pad_create.py diff --git a/tests/cryptpad/functional/test_spa_assets.py b/tests/cryptpad/custom/test_spa_assets.py similarity index 100% rename from tests/cryptpad/functional/test_spa_assets.py rename to tests/cryptpad/custom/test_spa_assets.py diff --git a/tests/custom-html-tiny/functional/test_serves_content.py b/tests/custom-html-tiny/custom/test_serves_content.py similarity index 100% rename from tests/custom-html-tiny/functional/test_serves_content.py rename to tests/custom-html-tiny/custom/test_serves_content.py diff --git a/tests/custom-html/PARITY.md b/tests/custom-html/PARITY.md index 870c877..f2d460b 100644 --- a/tests/custom-html/PARITY.md +++ b/tests/custom-html/PARITY.md @@ -1,13 +1,13 @@ # Parity — custom-html Phase-2 P2 mapping table: every `references/recipe-maintainer/recipe-info/custom-html/tests/*.py` has -a comparable cc-ci test under `tests/custom-html/functional/`, asserting the **same thing** (not just +a comparable cc-ci test under `tests/custom-html/custom/`, asserting the **same thing** (not just a renamed file). The Adversary cold-verifies parity by reading the source `recipe-info/` and the cc-ci file side-by-side. | recipe-maintainer file | cc-ci file | what's verified | status | |---|---|---|---| -| `recipe-info/custom-html/tests/health_check.py` | `tests/custom-html/functional/test_health_check.py` | The app is reachable over HTTPS and returns a successful response (the original asserted HTTP 200 against a persistent instance). The cc-ci port preserves the assertion shape — non-5xx status — and adapts to the ephemeral per-run domain via the `live_app` fixture. | **ported** | +| `recipe-info/custom-html/tests/health_check.py` | `tests/custom-html/custom/test_health_check.py` | The app is reachable over HTTPS and returns a successful response (the original asserted HTTP 200 against a persistent instance). The cc-ci port preserves the assertion shape — non-5xx status — and adapts to the ephemeral per-run domain via the `live_app` fixture. | **ported** | ## Recipe-specific tests (Phase-2 P3, ≥2 beyond parity) @@ -17,8 +17,8 @@ content, fetch it back"). Two new functional tests beyond parity: | cc-ci file | what's verified | rationale | |---|---|---| -| `tests/custom-html/functional/test_content_roundtrip.py` | Writes a uniquely-marked content file to the served volume via `lifecycle.exec_in_app` and asserts an HTTPS GET to the corresponding path returns that exact byte content — proves the app serves files written into its served volume, not a static synthetic page. | The recipe IS a content-server: a roundtrip is the canonical proof it works for what it's for. | -| `tests/custom-html/functional/test_content_type_header.py` | Writes both an `.html` and a `.txt` marker to the served volume, fetches each, and asserts `Content-Type` reflects the file type (`text/html`, `text/plain`) — proves nginx is properly serving with MIME-typed responses, not just returning bytes. | Distinctive nginx-served behavior — distinguishes a working nginx from a misconfigured one that emits everything as `application/octet-stream`. | +| `tests/custom-html/custom/test_content_roundtrip.py` | Writes a uniquely-marked content file to the served volume via `lifecycle.exec_in_app` and asserts an HTTPS GET to the corresponding path returns that exact byte content — proves the app serves files written into its served volume, not a static synthetic page. | The recipe IS a content-server: a roundtrip is the canonical proof it works for what it's for. | +| `tests/custom-html/custom/test_content_type_header.py` | Writes both an `.html` and a `.txt` marker to the served volume, fetches each, and asserts `Content-Type` reflects the file type (`text/html`, `text/plain`) — proves nginx is properly serving with MIME-typed responses, not just returning bytes. | Distinctive nginx-served behavior — distinguishes a working nginx from a misconfigured one that emits everything as `application/octet-stream`. | Both tests run in the **custom** stage against the same `live_app` shared deployment as the lifecycle overlays — no extra deploy, no extra teardown. @@ -32,7 +32,7 @@ via `lifecycle.exec_in_app` (volume-direct, immune to the post-backup serving ra ## Playwright (P6) -`tests/custom-html/playwright/test_browser_smoke.py` covers the browser-rendered nginx HTML — already +`tests/custom-html/custom/test_browser_smoke.py` covers the browser-rendered nginx HTML — already exercised inline by `tests/custom-html/test_install.py::test_serving_and_content` (lifecycle install overlay), which uses Playwright Chromium to confirm the page renders. The Phase-2 split file is the canonical home for browser-flow coverage and is invoked by the **custom** stage. diff --git a/tests/custom-html/playwright/test_browser_smoke.py b/tests/custom-html/custom/test_browser_smoke.py similarity index 93% rename from tests/custom-html/playwright/test_browser_smoke.py rename to tests/custom-html/custom/test_browser_smoke.py index b12a69c..0aac8cc 100644 --- a/tests/custom-html/playwright/test_browser_smoke.py +++ b/tests/custom-html/custom/test_browser_smoke.py @@ -1,7 +1,7 @@ """custom-html — Playwright UI flow (Phase 2 P6). -The recipe-maintainer corpus did not ship a Playwright test for custom-html — but plan §4.1 names -`playwright/` as the canonical home for browser flows where a recipe's core UX is a UI. custom-html +The recipe-maintainer corpus did not ship a Playwright test for custom-html — but the cfold layout +uses `custom/` as the canonical home for browser flows where a recipe's core UX is a UI. custom-html serves HTML; a browser-rendered fetch (vs raw HTTP) proves the page actually renders and any client- side resources resolve. Distinct from `tests/custom-html/test_install.py` which runs Playwright as part of the lifecycle INSTALL overlay; this file is the standalone Phase-2 custom-stage version, so a diff --git a/tests/custom-html/functional/test_content_roundtrip.py b/tests/custom-html/custom/test_content_roundtrip.py similarity index 100% rename from tests/custom-html/functional/test_content_roundtrip.py rename to tests/custom-html/custom/test_content_roundtrip.py diff --git a/tests/custom-html/functional/test_content_type_header.py b/tests/custom-html/custom/test_content_type_header.py similarity index 100% rename from tests/custom-html/functional/test_content_type_header.py rename to tests/custom-html/custom/test_content_type_header.py diff --git a/tests/custom-html/functional/test_health_check.py b/tests/custom-html/custom/test_health_check.py similarity index 100% rename from tests/custom-html/functional/test_health_check.py rename to tests/custom-html/custom/test_health_check.py diff --git a/tests/discourse/PARITY.md b/tests/discourse/PARITY.md index 18f6d25..b17b572 100644 --- a/tests/discourse/PARITY.md +++ b/tests/discourse/PARITY.md @@ -19,9 +19,9 @@ Defining behaviors exercised against the live per-run deploy: | cc-ci file | what's verified | rationale | |---|---|---| -| `functional/test_create_topic.py::test_create_topic_roundtrip` | Bootstraps an admin + API key via Rails in the `app` container (`_discourse.mint_admin`), POSTs `/posts.json` to create a NEW topic with a unique marker in title + body, then GETs `/t/.json` and asserts the title (Discourse `title_prettify`-aware) **and** the unique body marker round-tripped in the first post's `cooked`. | §4.3 "create the app's primary object — a topic — and read it back". Non-vacuous: the marker is unique per run, so a stale/echoed response can't pass; a wedged DB/Rails/posting path fails here even though `/srv/status` returns 200. | -| `functional/test_site_basic.py::test_site_json_has_discourse_config` | GETs `/site.json` and asserts a Discourse-specific config structure (e.g. a `categories` list), not a bare 200. | Proves Rails is serving its real site config JSON (a distinctive Discourse structure), distinguishing "the forum backend is up + emitting its API" from "a static/error page at /". | -| `functional/test_health_check.py::test_discourse_srv_status_ok` | GETs `/srv/status` and asserts the Discourse readiness signal (Rails serving). | Baseline readiness (parity-aligned health check). | +| `custom/test_create_topic.py::test_create_topic_roundtrip` | Bootstraps an admin + API key via Rails in the `app` container (`_discourse.mint_admin`), POSTs `/posts.json` to create a NEW topic with a unique marker in title + body, then GETs `/t/.json` and asserts the title (Discourse `title_prettify`-aware) **and** the unique body marker round-tripped in the first post's `cooked`. | §4.3 "create the app's primary object — a topic — and read it back". Non-vacuous: the marker is unique per run, so a stale/echoed response can't pass; a wedged DB/Rails/posting path fails here even though `/srv/status` returns 200. | +| `custom/test_site_basic.py::test_site_json_has_discourse_config` | GETs `/site.json` and asserts a Discourse-specific config structure (e.g. a `categories` list), not a bare 200. | Proves Rails is serving its real site config JSON (a distinctive Discourse structure), distinguishing "the forum backend is up + emitting its API" from "a static/error page at /". | +| `custom/test_health_check.py::test_discourse_srv_status_ok` | GETs `/srv/status` and asserts the Discourse readiness signal (Rails serving). | Baseline readiness (parity-aligned health check). | Two recipe-specific functional tests (create-topic round-trip + site.json config) + the health check = the ≥2 floor met, with a real create-an-object + read-it-back as the characteristic-behavior test. diff --git a/tests/discourse/functional/_discourse.py b/tests/discourse/custom/_discourse.py similarity index 100% rename from tests/discourse/functional/_discourse.py rename to tests/discourse/custom/_discourse.py diff --git a/tests/discourse/functional/test_create_topic.py b/tests/discourse/custom/test_create_topic.py similarity index 100% rename from tests/discourse/functional/test_create_topic.py rename to tests/discourse/custom/test_create_topic.py diff --git a/tests/discourse/functional/test_health_check.py b/tests/discourse/custom/test_health_check.py similarity index 100% rename from tests/discourse/functional/test_health_check.py rename to tests/discourse/custom/test_health_check.py diff --git a/tests/discourse/functional/test_site_basic.py b/tests/discourse/custom/test_site_basic.py similarity index 100% rename from tests/discourse/functional/test_site_basic.py rename to tests/discourse/custom/test_site_basic.py diff --git a/tests/drone/functional/__init__.py b/tests/drone/custom/__init__.py similarity index 100% rename from tests/drone/functional/__init__.py rename to tests/drone/custom/__init__.py diff --git a/tests/drone/functional/test_scm_configured.py b/tests/drone/custom/test_scm_configured.py similarity index 100% rename from tests/drone/functional/test_scm_configured.py rename to tests/drone/custom/test_scm_configured.py diff --git a/tests/drone/install_steps.sh b/tests/drone/install_steps.sh index d920042..5099c84 100755 --- a/tests/drone/install_steps.sh +++ b/tests/drone/install_steps.sh @@ -10,7 +10,7 @@ # 4. Sets DRONE_USER_CREATE so the gitea ci_admin becomes drone's first admin on login. # # If the deps file is absent or has no gitea entry, drone is still deployed (without SCM wiring); -# the functional/test_scm_configured.py test then FAILS, which is the correct signal. +# the custom/test_scm_configured.py test then FAILS, which is the correct signal. # # Env supplied by the harness: # CCCI_APP_DOMAIN — the per-run drone app domain diff --git a/tests/ghost/PARITY.md b/tests/ghost/PARITY.md index 738d749..952d55e 100644 --- a/tests/ghost/PARITY.md +++ b/tests/ghost/PARITY.md @@ -11,15 +11,15 @@ and a JSON Content/Admin API at `/ghost/api/*`. Defining behaviors exercised: | cc-ci file | what's verified | rationale | |---|---|---| -| `tests/ghost/functional/test_content_api.py` | GETs `/ghost/api/content/settings/`; asserts 200 with `{"settings": {...}}` envelope OR 401/403 with a Ghost error envelope. | Distinguishes "the ghost-server JS process is up + emitting its API" from "a static themed page is served at /." A wedged Ghost backend → 5xx; misrouted nginx → 404. | -| `tests/ghost/functional/test_admin_redirect.py` | GETs `/ghost/`; asserts 200 or 302 + Ghost branding/SPA references in the response (or a redirect to /ghost/#/setup on fresh deploy). | Proves the admin route is wired through the nginx proxy. Distinguishes "admin SPA bound" from "404 (route missing)" or "5xx (broken)." | +| `tests/ghost/custom/test_content_api.py` | GETs `/ghost/api/content/settings/`; asserts 200 with `{"settings": {...}}` envelope OR 401/403 with a Ghost error envelope. | Distinguishes "the ghost-server JS process is up + emitting its API" from "a static themed page is served at /." A wedged Ghost backend → 5xx; misrouted nginx → 404. | +| `tests/ghost/custom/test_admin_redirect.py` | GETs `/ghost/`; asserts 200 or 302 + Ghost branding/SPA references in the response (or a redirect to /ghost/#/setup on fresh deploy). | Proves the admin route is wired through the nginx proxy. Distinguishes "admin SPA bound" from "404 (route missing)" or "5xx (broken)." | Two specific tests + parity health_check = ≥2 floor met. ## Plan §4.3 prescribed deeper test — AUTHORED (closes DEFERRED ghost create-post) §4.3 named "create-a-post round-trip" for ghost. Implemented in -`tests/ghost/functional/test_post_roundtrip.py` (helper `functional/_ghost.py`): +`tests/ghost/custom/test_post_roundtrip.py` (helper `custom/_ghost.py`): 1. Wait for the Admin API healthcheck (`GET /ghost/api/admin/site/` → 200). 2. Setup the Ghost owner (POST `/ghost/api/admin/authentication/setup/`, fresh deploy) + establish an admin **session cookie** (POST `/ghost/api/admin/session/`) — cookie-aware stdlib opener, diff --git a/tests/ghost/functional/_ghost.py b/tests/ghost/custom/_ghost.py similarity index 100% rename from tests/ghost/functional/_ghost.py rename to tests/ghost/custom/_ghost.py diff --git a/tests/ghost/functional/test_admin_redirect.py b/tests/ghost/custom/test_admin_redirect.py similarity index 100% rename from tests/ghost/functional/test_admin_redirect.py rename to tests/ghost/custom/test_admin_redirect.py diff --git a/tests/ghost/functional/test_content_api.py b/tests/ghost/custom/test_content_api.py similarity index 100% rename from tests/ghost/functional/test_content_api.py rename to tests/ghost/custom/test_content_api.py diff --git a/tests/ghost/functional/test_health_check.py b/tests/ghost/custom/test_health_check.py similarity index 100% rename from tests/ghost/functional/test_health_check.py rename to tests/ghost/custom/test_health_check.py diff --git a/tests/ghost/functional/test_post_roundtrip.py b/tests/ghost/custom/test_post_roundtrip.py similarity index 100% rename from tests/ghost/functional/test_post_roundtrip.py rename to tests/ghost/custom/test_post_roundtrip.py diff --git a/tests/ghost/recipe_meta.py b/tests/ghost/recipe_meta.py index 44b2580..8296c4a 100644 --- a/tests/ghost/recipe_meta.py +++ b/tests/ghost/recipe_meta.py @@ -2,7 +2,7 @@ # Ghost serves an HTML site at `/`; admin UI at `/ghost/`. The first GET to /ghost/ redirects # to the setup wizard (302). Ghost exposes a JSON Content API at /ghost/api/content/ which # requires an API key; the Admin API at /ghost/api/admin/ requires a session/token (see -# functional/_ghost.py — version-negotiated, no /v3/ path). +# custom/_ghost.py — version-negotiated, no /v3/ path). # State lives in a **MySQL** `ghost` DB (compose `db` service, mysql:8.0) + the `ghost_content` # volume (themes/images) — NOT sqlite. The `db` service is backupbot-labelled with a logical # mysqldump pre-hook; P4 (ops.py + test_{backup,restore,upgrade}.py) seeds a `ci_marker` row there. diff --git a/tests/hedgedoc/PARITY.md b/tests/hedgedoc/PARITY.md index 3df80c5..d3660ee 100644 --- a/tests/hedgedoc/PARITY.md +++ b/tests/hedgedoc/PARITY.md @@ -14,8 +14,8 @@ HedgeDoc's defining behaviors: | cc-ci file | what's verified | rationale | |---|---|---| -| `tests/hedgedoc/functional/test_health_check.py` | `GET /` → 200 or 302 | Proves the app is up and routing through Traefik. A wedged HedgeDoc returns 5xx or no response. | -| `tests/hedgedoc/functional/test_branding.py` | `GET /` HTML contains hedgedoc/codimd/hackmd markers OR bundle asset refs | Distinguishes "HedgeDoc is serving its own content" from "fallback page." A misrouted or empty backend lacks these markers. | +| `tests/hedgedoc/custom/test_health_check.py` | `GET /` → 200 or 302 | Proves the app is up and routing through Traefik. A wedged HedgeDoc returns 5xx or no response. | +| `tests/hedgedoc/custom/test_branding.py` | `GET /` HTML contains hedgedoc/codimd/hackmd markers OR bundle asset refs | Distinguishes "HedgeDoc is serving its own content" from "fallback page." A misrouted or empty backend lacks these markers. | ## Backup data-integrity diff --git a/tests/hedgedoc/functional/test_branding.py b/tests/hedgedoc/custom/test_branding.py similarity index 100% rename from tests/hedgedoc/functional/test_branding.py rename to tests/hedgedoc/custom/test_branding.py diff --git a/tests/hedgedoc/functional/test_health_check.py b/tests/hedgedoc/custom/test_health_check.py similarity index 100% rename from tests/hedgedoc/functional/test_health_check.py rename to tests/hedgedoc/custom/test_health_check.py diff --git a/tests/immich/PARITY.md b/tests/immich/PARITY.md index b4e1546..9859b8c 100644 --- a/tests/immich/PARITY.md +++ b/tests/immich/PARITY.md @@ -5,7 +5,7 @@ Reference corpus: `references/recipe-maintainer/recipe-info/immich/tests/` (heal ## Parity ports | recipe-maintainer test | cc-ci test | what's verified | |---|---|---| -| `health_check.py` | `tests/immich/functional/test_health_check.py::test_immich_returns_200` | HTTP 200/301/302 from `/` (immich web SPA served). | +| `health_check.py` | `tests/immich/custom/test_health_check.py::test_immich_returns_200` | HTTP 200/301/302 from `/` (immich web SPA served). | ## Recipe-specific functional tests (P3, ≥2 separate tests — characteristic behavior) 1. **`test_asset_upload.py::test_immich_upload_asset_readback_and_thumbnail`** — the §4.3 diff --git a/tests/immich/functional/test_asset_processing.py b/tests/immich/custom/test_asset_processing.py similarity index 100% rename from tests/immich/functional/test_asset_processing.py rename to tests/immich/custom/test_asset_processing.py diff --git a/tests/immich/functional/test_asset_upload.py b/tests/immich/custom/test_asset_upload.py similarity index 100% rename from tests/immich/functional/test_asset_upload.py rename to tests/immich/custom/test_asset_upload.py diff --git a/tests/immich/functional/test_health_check.py b/tests/immich/custom/test_health_check.py similarity index 100% rename from tests/immich/functional/test_health_check.py rename to tests/immich/custom/test_health_check.py diff --git a/tests/keycloak/PARITY.md b/tests/keycloak/PARITY.md index 2b52a82..3bb835d 100644 --- a/tests/keycloak/PARITY.md +++ b/tests/keycloak/PARITY.md @@ -5,8 +5,8 @@ Phase-2 P2 mapping table. The Adversary cold-verifies parity by reading the sour | recipe-maintainer file | cc-ci file | what's verified | status | |---|---|---|---| -| `recipe-info/keycloak/tests/health_check.py` | `tests/keycloak/functional/test_health_check.py` | The keycloak master realm endpoint (`/realms/master`) returns HTTP 200 — the original's assertion shape, preserved. The cc-ci port adapts to the ephemeral per-run domain via the `live_app` fixture. | **ported** | -| `recipe-info/keycloak/tests/oidc_integration.py` | (deferred to Q3 lasuite-docs) | The original is a **cross-recipe** integration test: it expects both keycloak AND lasuite-docs deployed, with a pre-seeded credentials TOML, and proves a keycloak-issued token is accepted by lasuite-docs. This requires the Phase-2 dependency resolver (Q0.4/Q2.3) + lasuite-docs Phase-2 enrollment. Will land as `tests/lasuite-docs/functional/test_oidc_with_keycloak.py` in Q3, sharing the SSO-setup harness. | **deferred to Q3** (logged in DECISIONS.md) | +| `recipe-info/keycloak/tests/health_check.py` | `tests/keycloak/custom/test_health_check.py` | The keycloak master realm endpoint (`/realms/master`) returns HTTP 200 — the original's assertion shape, preserved. The cc-ci port adapts to the ephemeral per-run domain via the `live_app` fixture. | **ported** | +| `recipe-info/keycloak/tests/oidc_integration.py` | (deferred to Q3 lasuite-docs) | The original is a **cross-recipe** integration test: it expects both keycloak AND lasuite-docs deployed, with a pre-seeded credentials TOML, and proves a keycloak-issued token is accepted by lasuite-docs. This requires the Phase-2 dependency resolver (Q0.4/Q2.3) + lasuite-docs Phase-2 enrollment. Will land as `tests/lasuite-docs/custom/test_oidc_with_keycloak.py` in Q3, sharing the SSO-setup harness. | **deferred to Q3** (logged in DECISIONS.md) | ## Recipe-specific tests (Phase-2 P3, ≥2 beyond parity) @@ -16,8 +16,8 @@ marker realm across upgrade/backup/restore — Phase 1d/1e). Two new functional | cc-ci file | what's verified | rationale | |---|---|---| -| `tests/keycloak/functional/test_password_grant_token.py` | Obtains an admin-CLI access token via the password grant (`grant_type=password`) against `/realms/master/protocol/openid-connect/token`, asserts the token is a valid JWT (3 base64url-encoded segments), decodes the payload, and asserts the JWT claims include `iss` matching the live domain, `azp == "admin-cli"`, `typ == "Bearer"`, and a future `exp`. | The defining keycloak behavior is issuing JWTs; this test does the canonical password-grant flow against the real running keycloak (real admin user, real password from the abra-generated secret) and proves the JWT contract is intact. Non-vacuous: a wrongly-configured realm, broken signing key, or wrong issuer would fail the claim assertions. | -| `tests/keycloak/functional/test_create_client_and_use.py` | Authenticates as admin → creates a confidential client in the master realm via admin API with a known `clientId` + a known client secret → obtains a token via `grant_type=client_credentials` for that client → asserts the token's `azp` (authorized party) matches the new client's clientId → deletes the client (idempotent cleanup). | Proves the full lifecycle of admin-API client creation + service-account token issuance, the canonical "real app integrating with keycloak" flow. Non-vacuous: tests TWO grant types (password + client-credentials) and the admin-API CRUD on clients. | +| `tests/keycloak/custom/test_password_grant_token.py` | Obtains an admin-CLI access token via the password grant (`grant_type=password`) against `/realms/master/protocol/openid-connect/token`, asserts the token is a valid JWT (3 base64url-encoded segments), decodes the payload, and asserts the JWT claims include `iss` matching the live domain, `azp == "admin-cli"`, `typ == "Bearer"`, and a future `exp`. | The defining keycloak behavior is issuing JWTs; this test does the canonical password-grant flow against the real running keycloak (real admin user, real password from the abra-generated secret) and proves the JWT contract is intact. Non-vacuous: a wrongly-configured realm, broken signing key, or wrong issuer would fail the claim assertions. | +| `tests/keycloak/custom/test_create_client_and_use.py` | Authenticates as admin → creates a confidential client in the master realm via admin API with a known `clientId` + a known client secret → obtains a token via `grant_type=client_credentials` for that client → asserts the token's `azp` (authorized party) matches the new client's clientId → deletes the client (idempotent cleanup). | Proves the full lifecycle of admin-API client creation + service-account token issuance, the canonical "real app integrating with keycloak" flow. Non-vacuous: tests TWO grant types (password + client-credentials) and the admin-API CRUD on clients. | Both tests run in the **custom** tier against the same `live_app` shared deployment as the lifecycle overlays — no extra deploy, no extra teardown. diff --git a/tests/keycloak/functional/test_create_client_and_use.py b/tests/keycloak/custom/test_create_client_and_use.py similarity index 99% rename from tests/keycloak/functional/test_create_client_and_use.py rename to tests/keycloak/custom/test_create_client_and_use.py index 488ff8a..62771f1 100644 --- a/tests/keycloak/functional/test_create_client_and_use.py +++ b/tests/keycloak/custom/test_create_client_and_use.py @@ -28,7 +28,7 @@ import urllib.parse import urllib.request import uuid -# kc_admin.py lives in tests/keycloak/, one level up from this file in tests/keycloak/functional/ +# kc_admin.py lives in tests/keycloak/, one level up from this file in tests/keycloak/custom/ sys.path.insert(0, os.path.join(os.path.dirname(__file__), "..")) sys.path.insert(0, os.path.join(os.path.dirname(__file__), "..", "..", "..", "runner")) import kc_admin # noqa: E402 diff --git a/tests/keycloak/functional/test_health_check.py b/tests/keycloak/custom/test_health_check.py similarity index 100% rename from tests/keycloak/functional/test_health_check.py rename to tests/keycloak/custom/test_health_check.py diff --git a/tests/keycloak/functional/test_password_grant_token.py b/tests/keycloak/custom/test_password_grant_token.py similarity index 98% rename from tests/keycloak/functional/test_password_grant_token.py rename to tests/keycloak/custom/test_password_grant_token.py index b8b0efe..44b6d17 100644 --- a/tests/keycloak/functional/test_password_grant_token.py +++ b/tests/keycloak/custom/test_password_grant_token.py @@ -18,7 +18,7 @@ import os import sys import time -# kc_admin.py lives in tests/keycloak/, one level up from this file in tests/keycloak/functional/ +# kc_admin.py lives in tests/keycloak/, one level up from this file in tests/keycloak/custom/ sys.path.insert(0, os.path.join(os.path.dirname(__file__), "..")) sys.path.insert(0, os.path.join(os.path.dirname(__file__), "..", "..", "..", "runner")) import kc_admin # noqa: E402 diff --git a/tests/lasuite-docs/PARITY.md b/tests/lasuite-docs/PARITY.md index 0448a99..c3c54aa 100644 --- a/tests/lasuite-docs/PARITY.md +++ b/tests/lasuite-docs/PARITY.md @@ -5,16 +5,16 @@ Phase-2 P2 mapping table. The Adversary cold-verifies parity by reading the sour | recipe-maintainer file | cc-ci file | what's verified | status | |---|---|---|---| -| `recipe-info/lasuite-docs/tests/health_check.py` | `tests/lasuite-docs/functional/test_health_check.py` | The app serves over HTTPS and returns a successful response (200/301/302). The cc-ci port preserves the assertion shape, adapted to the ephemeral per-run domain via `live_app`. | **ported** | -| `recipe-info/lasuite-docs/tests/oidc_login.py` | `tests/lasuite-docs/functional/test_oidc_with_keycloak.py` (Q2.4 acceptance, partial port) + `test_auth_required.py` (proves the gate is wired) | The original's flow: deploy keycloak + setup realm/client/user + obtain JWT + use it against lasuite-docs's protected API. The cc-ci pair: (a) `test_oidc_with_keycloak` deploys keycloak as a Q2.3 dep, sets up realm/client/user, obtains a real JWT, validates iss/azp/typ/exp claims; (b) `test_auth_required` proves lasuite-docs's backend API requires auth (401). Step-(c) — actually USING the JWT against lasuite-docs — requires wiring the dep keycloak's client_secret + OIDC env into lasuite-docs's `.env` at install time; **see "Deferred (Q3.1 follow-up)" below**. | **partial; see follow-up** | +| `recipe-info/lasuite-docs/tests/health_check.py` | `tests/lasuite-docs/custom/test_health_check.py` | The app serves over HTTPS and returns a successful response (200/301/302). The cc-ci port preserves the assertion shape, adapted to the ephemeral per-run domain via `live_app`. | **ported** | +| `recipe-info/lasuite-docs/tests/oidc_login.py` | `tests/lasuite-docs/custom/test_oidc_with_keycloak.py` (Q2.4 acceptance, partial port) + `test_auth_required.py` (proves the gate is wired) | The original's flow: deploy keycloak + setup realm/client/user + obtain JWT + use it against lasuite-docs's protected API. The cc-ci pair: (a) `test_oidc_with_keycloak` deploys keycloak as a Q2.3 dep, sets up realm/client/user, obtains a real JWT, validates iss/azp/typ/exp claims; (b) `test_auth_required` proves lasuite-docs's backend API requires auth (401). Step-(c) — actually USING the JWT against lasuite-docs — requires wiring the dep keycloak's client_secret + OIDC env into lasuite-docs's `.env` at install time; **see "Deferred (Q3.1 follow-up)" below**. | **partial; see follow-up** | | `recipe-info/lasuite-docs/tests/upload_conversion.py` | (Q3.1 follow-up — needs OIDC env wired into lasuite-docs first) | The original uploads .md + .docx via authenticated `POST /api/v1.0/documents//upload` and asserts the y-provider + docspec conversion paths fire. The cc-ci port requires authentication, which requires OIDC env wiring (see below). | **deferred** | ## Recipe-specific tests (Phase-2 P3, ≥2 beyond parity) | cc-ci file | what's verified | rationale | |---|---|---| -| `tests/lasuite-docs/functional/test_oidc_with_keycloak.py` | Deploys keycloak as a per-run **dep** (Q2.3 resolver via `DEPS = ["keycloak"]`), sets up a realm/client/user, exercises the OIDC discovery endpoint + the password-grant flow against the dep keycloak, validates the returned JWT's iss/azp/typ/exp claims. | The recipe is **OIDC-dependent** by design; proving the SSO provider deploys + issues tokens + the JWT contract is intact is a defining lasuite-docs behavior (and the Q2 gate acceptance test). | -| `tests/lasuite-docs/functional/test_auth_required.py` | GETs `/api/v1.0/users/me/` without a token; asserts **401 Unauthorized** (or 403). Non-vacuous: distinguishes a correctly-wired OIDC gate (401) from anonymous access (200), missing route (404), and broken backend (5xx). | Proves lasuite-docs's **own** auth posture (distinct from the SSO provider's token issuance). Together with `test_oidc_with_keycloak` this exercises both sides of the OIDC flow's plumbing. | +| `tests/lasuite-docs/custom/test_oidc_with_keycloak.py` | Deploys keycloak as a per-run **dep** (Q2.3 resolver via `DEPS = ["keycloak"]`), sets up a realm/client/user, exercises the OIDC discovery endpoint + the password-grant flow against the dep keycloak, validates the returned JWT's iss/azp/typ/exp claims. | The recipe is **OIDC-dependent** by design; proving the SSO provider deploys + issues tokens + the JWT contract is intact is a defining lasuite-docs behavior (and the Q2 gate acceptance test). | +| `tests/lasuite-docs/custom/test_auth_required.py` | GETs `/api/v1.0/users/me/` without a token; asserts **401 Unauthorized** (or 403). Non-vacuous: distinguishes a correctly-wired OIDC gate (401) from anonymous access (200), missing route (404), and broken backend (5xx). | Proves lasuite-docs's **own** auth posture (distinct from the SSO provider's token issuance). Together with `test_oidc_with_keycloak` this exercises both sides of the OIDC flow's plumbing. | Two specific tests — the ≥2 floor is met. Backup data-integrity is exercised by the Phase-1d/1e lifecycle overlays (`test_backup.py`/`test_restore.py` + `ops.py`). @@ -30,7 +30,7 @@ with a real OIDC-issued JWT. To round these out, the cc-ci side needs: - Inserts `SECRET_OIDC_RPCS_VERSION=v1` + the secret value via `abra app secret insert`. - Appends to lasuite-docs's `.env`: `OIDC_REALM`, `OIDC_CLIENT_ID`, `OIDC_OP_*` URLs pointing at the dep keycloak. -2. **Authenticated test**: a new `tests/lasuite-docs/functional/test_create_doc.py` performs the +2. **Authenticated test**: a new `tests/lasuite-docs/custom/test_create_doc.py` performs the password grant against the dep keycloak, presents the JWT to lasuite-docs's `POST /api/v1.0/documents/` (create a doc), asserts the doc is fetched back via `GET /api/v1.0/documents//` — the §4.3 prescribed create-and-read-back. diff --git a/tests/lasuite-docs/functional/test_auth_required.py b/tests/lasuite-docs/custom/test_auth_required.py similarity index 100% rename from tests/lasuite-docs/functional/test_auth_required.py rename to tests/lasuite-docs/custom/test_auth_required.py diff --git a/tests/lasuite-docs/functional/test_create_doc.py b/tests/lasuite-docs/custom/test_create_doc.py similarity index 100% rename from tests/lasuite-docs/functional/test_create_doc.py rename to tests/lasuite-docs/custom/test_create_doc.py diff --git a/tests/lasuite-docs/functional/test_health_check.py b/tests/lasuite-docs/custom/test_health_check.py similarity index 100% rename from tests/lasuite-docs/functional/test_health_check.py rename to tests/lasuite-docs/custom/test_health_check.py diff --git a/tests/lasuite-docs/functional/test_oidc_login.py b/tests/lasuite-docs/custom/test_oidc_login.py similarity index 100% rename from tests/lasuite-docs/functional/test_oidc_login.py rename to tests/lasuite-docs/custom/test_oidc_login.py diff --git a/tests/lasuite-docs/functional/test_oidc_with_keycloak.py b/tests/lasuite-docs/custom/test_oidc_with_keycloak.py similarity index 100% rename from tests/lasuite-docs/functional/test_oidc_with_keycloak.py rename to tests/lasuite-docs/custom/test_oidc_with_keycloak.py diff --git a/tests/lasuite-docs/install_steps.sh b/tests/lasuite-docs/install_steps.sh index d845418..85e182e 100755 --- a/tests/lasuite-docs/install_steps.sh +++ b/tests/lasuite-docs/install_steps.sh @@ -1,6 +1,6 @@ #!/usr/bin/env bash # lasuite-docs — INSTALL-TIME OIDC wiring hook (rcust P2b; migrated from the deleted -# setup_custom_tests.sh post-deploy path — sibling of lasuite-drive/-meet's hooks). +# old post-deploy setup path — sibling of lasuite-drive/-meet's hooks). # # Runs during the install tier AFTER `abra app new` + EXTRA_ENV + `abra app secret generate`, and # BEFORE the single `abra app deploy` (lifecycle.py::_run_install_steps). Writing OIDC env + the diff --git a/tests/lasuite-docs/recipe_meta.py b/tests/lasuite-docs/recipe_meta.py index 6cdccb3..b91112d 100644 --- a/tests/lasuite-docs/recipe_meta.py +++ b/tests/lasuite-docs/recipe_meta.py @@ -10,7 +10,7 @@ HTTP_TIMEOUT = 600 # Phase 2 Q2.3 deps: lasuite-docs's recipe-maintainer corpus declares `requires = ["keycloak"]`. # Declaring it here makes the orchestrator deploy a per-run keycloak BEFORE lasuite-docs so the -# OIDC-flow functional test (`functional/test_oidc_with_keycloak.py`) can run against a real +# OIDC-flow custom test (`custom/test_oidc_with_keycloak.py`) can run against a real # provider in the same run. The dep is undeployed AFTER the parent in the orchestrator's `finally`. DEPS = ["keycloak"] diff --git a/tests/lasuite-drive/PARITY.md b/tests/lasuite-drive/PARITY.md index d2667f9..af0163b 100644 --- a/tests/lasuite-drive/PARITY.md +++ b/tests/lasuite-drive/PARITY.md @@ -5,16 +5,16 @@ Phase-2 P2 mapping table. The Adversary cold-verifies parity by reading the sour **Enrollment status:** Q3.2 SSO iteration. Base deploy + lifecycle (install/upgrade/backup/restore data-integrity) + parity health_check landed first; the base proved cold-green @2026-05-28 (all 12 -services incl. onlyoffice+collabora). Now landed on top: `DEPS=["keycloak"]` + `setup_custom_tests.sh` -OIDC wiring + the OIDC SSO test + the MinIO storage round-trip (the §4.3 specifics). WOPI discovery is +services incl. onlyoffice+collabora). Now landed on top: `DEPS=["keycloak"]` + install-time OIDC +wiring + the OIDC SSO test + the MinIO storage round-trip (the §4.3 specifics). WOPI discovery is a further (3rd) test beyond the ≥2 floor — still planned. This file is updated as each row lands; nothing is a silent omission. | recipe-maintainer file | cc-ci file | what's verified | status | |---|---|---|---| -| `recipe-info/lasuite-drive/tests/health_check.py` | `tests/lasuite-drive/functional/test_health_check.py` | App serves over HTTPS and returns 200/301/302 from `/`. Port preserves the assertion shape, adapted to the ephemeral per-run domain via `live_app`. | **ported** | -| `recipe-info/lasuite-drive/tests/oidc_login.py` | `tests/lasuite-drive/functional/test_oidc_with_keycloak.py` | Original: Drive `/api/v1.0/authenticate/` redirects to Keycloak → password-grant token → `/api/v1.0/users/me/` returns the user. cc-ci port deploys keycloak as a per-run dep (`DEPS=["keycloak"]`), wires OIDC env via `setup_custom_tests.sh`, exercises discovery + password grant + JWT claims (iss/azp/typ/exp) against the dep realm `lasuite-drive` (mirrors the proven lasuite-docs `test_oidc_with_keycloak`). `@requires_deps` so a deps-not-ready skip fails the run (F2-11), not a silent green. | **ported** | -| `recipe-info/lasuite-drive/tests/wopi_configured.py` | `tests/lasuite-drive/functional/test_wopi_configured.py` (planned) | Original: Collabora + OnlyOffice WOPI discovery endpoints return valid WOPI XML. cc-ci port checks the Collabora discovery XML over the flattened `collabora-` route (pure HTTP, no browser/SSO). | **pending** | +| `recipe-info/lasuite-drive/tests/health_check.py` | `tests/lasuite-drive/custom/test_health_check.py` | App serves over HTTPS and returns 200/301/302 from `/`. Port preserves the assertion shape, adapted to the ephemeral per-run domain via `live_app`. | **ported** | +| `recipe-info/lasuite-drive/tests/oidc_login.py` | `tests/lasuite-drive/custom/test_oidc_with_keycloak.py` | Original: Drive `/api/v1.0/authenticate/` redirects to Keycloak → password-grant token → `/api/v1.0/users/me/` returns the user. cc-ci port deploys keycloak as a per-run dep (`DEPS=["keycloak"]`), wires OIDC env at install time, exercises discovery + password grant + JWT claims (iss/azp/typ/exp) against the dep realm `lasuite-drive` (mirrors the proven lasuite-docs `test_oidc_with_keycloak`). `@requires_deps` so a deps-not-ready skip fails the run (F2-11), not a silent green. | **ported** | +| `recipe-info/lasuite-drive/tests/wopi_configured.py` | `tests/lasuite-drive/custom/test_wopi_configured.py` (planned) | Original: Collabora + OnlyOffice WOPI discovery endpoints return valid WOPI XML. cc-ci port checks the Collabora discovery XML over the flattened `collabora-` route (pure HTTP, no browser/SSO). | **pending** | | `recipe-info/lasuite-drive/tests/wopi_on_startup.py` | (see DECISIONS / DEFERRED) | Original: greps celery worker container logs for the entrypoint WOPI trigger. cc-ci port via `docker service logs` on the celery service. | **pending** | | `recipe-info/lasuite-drive/tests/celery_beat_wopi.py` | (likely DEFERRED — "thorough mode only") | Original sleeps 15–90s waiting for Celery Beat to fire; recipe-maintainer marks it "thorough mode only". Candidate for the `--extra-tests` opt-in (DEFERRED.md), like the matrix-synapse operational ports. | **likely deferred** | @@ -22,9 +22,9 @@ nothing is a silent omission. | cc-ci file | what's verified | status | |---|---|---| -| `functional/test_oidc_with_keycloak.py` | SSO round-trip against the dep keycloak: OIDC discovery advertises realm `lasuite-drive`; password grant yields a valid JWT with iss/azp/typ/exp claims. Drive is OIDC-required — this is its defining auth path. | **landed** | -| `functional/test_minio_storage.py` | The §4.3 create-an-object + read-it-back, at Drive's storage layer: confirms the `drive-media-storage` MinIO bucket exists, then a real upload → list → download round-trip (unique marker) asserting the bytes survive. Runs `mc` inside the `minio` container with the in-container root creds. Non-health-only: a missing bucket or broken object store fails it. | **landed** | -| `functional/test_wopi_configured.py` (planned, 3rd beyond floor) | Collabora WOPI discovery XML served + valid over the flattened `collabora-` route — Drive's in-browser office-editing feature. | **planned** | +| `custom/test_oidc_with_keycloak.py` | SSO round-trip against the dep keycloak: OIDC discovery advertises realm `lasuite-drive`; password grant yields a valid JWT with iss/azp/typ/exp claims. Drive is OIDC-required — this is its defining auth path. | **landed** | +| `custom/test_minio_storage.py` | The §4.3 create-an-object + read-it-back, at Drive's storage layer: confirms the `drive-media-storage` MinIO bucket exists, then a real upload → list → download round-trip (unique marker) asserting the bytes survive. Runs `mc` inside the `minio` container with the in-container root creds. Non-health-only: a missing bucket or broken object store fails it. | **landed** | +| `custom/test_wopi_configured.py` (planned, 3rd beyond floor) | Collabora WOPI discovery XML served + valid over the flattened `collabora-` route — Drive's in-browser office-editing feature. | **planned** | ## Backup data-integrity (P4) — landed diff --git a/tests/lasuite-drive/functional/test_health_check.py b/tests/lasuite-drive/custom/test_health_check.py similarity index 100% rename from tests/lasuite-drive/functional/test_health_check.py rename to tests/lasuite-drive/custom/test_health_check.py diff --git a/tests/lasuite-drive/functional/test_minio_storage.py b/tests/lasuite-drive/custom/test_minio_storage.py similarity index 100% rename from tests/lasuite-drive/functional/test_minio_storage.py rename to tests/lasuite-drive/custom/test_minio_storage.py diff --git a/tests/lasuite-drive/functional/test_oidc_with_keycloak.py b/tests/lasuite-drive/custom/test_oidc_with_keycloak.py similarity index 97% rename from tests/lasuite-drive/functional/test_oidc_with_keycloak.py rename to tests/lasuite-drive/custom/test_oidc_with_keycloak.py index 7a8f8f7..4364026 100644 --- a/tests/lasuite-drive/functional/test_oidc_with_keycloak.py +++ b/tests/lasuite-drive/custom/test_oidc_with_keycloak.py @@ -12,7 +12,7 @@ Mirrors the proven lasuite-docs SSO model: `deps-not-ready` reason — and (per F2-11) the orchestrator then fails the run rather than going green on a skipped SSO test. -SOURCE: adapted from tests/lasuite-docs/functional/test_oidc_with_keycloak.py (Q2.4 acceptance). +SOURCE: adapted from tests/lasuite-docs/custom/test_oidc_with_keycloak.py (Q2.4 acceptance). """ from __future__ import annotations diff --git a/tests/lasuite-drive/ops.py b/tests/lasuite-drive/ops.py index 2b567fb..279d347 100644 --- a/tests/lasuite-drive/ops.py +++ b/tests/lasuite-drive/ops.py @@ -14,14 +14,14 @@ from harness import lifecycle # noqa: E402 def pre_install(ctx): - """Post-deploy seed for the custom tier (the former setup_custom_tests.sh, moved here in rcust + """Post-deploy seed for the custom tier (the former post-deploy setup hook, moved here in rcust P2b — install_steps.sh runs PRE-deploy and cannot touch the live stack). The deploy alone does NOT create the MinIO bucket: `minio-createbuckets` is a `replicas:0` one-shot (restart_policy: none) that must be triggered. The MinIO storage test asserts the bucket exists, so trigger it here and poll. `--detach` is REQUIRED: the job creates the bucket then EXITS 0, so it never holds a steady 1/1 replica — a blocking scale would wait forever. - BEST-EFFORT, like the setup_custom_tests.sh it replaced: on poll timeout we WARN and continue + BEST-EFFORT, like the old post-deploy hook it replaced: on poll timeout we WARN and continue (the one-shot often lands just after the window). The custom-tier MinIO storage test is the real gate for a genuinely missing bucket — failing the install op here was an rcust M2 regression (the original hook fell through on timeout by design).""" diff --git a/tests/lasuite-drive/recipe_meta.py b/tests/lasuite-drive/recipe_meta.py index d444953..4dea90c 100644 --- a/tests/lasuite-drive/recipe_meta.py +++ b/tests/lasuite-drive/recipe_meta.py @@ -20,7 +20,7 @@ HTTP_TIMEOUT = 900 # Base deploy/lifecycle proven cold-green @2026-05-28 (install: pass; 12 services incl. # onlyoffice+collabora) once the Docker Hub rate limit was fixed. Declaring DEPS makes the # orchestrator provision keycloak (realm/client/user) BEFORE the single deploy; -# functional/test_oidc_with_keycloak.py then exercises the SSO flow. +# custom/test_oidc_with_keycloak.py then exercises the SSO flow. DEPS = ["keycloak"] # OIDC is wired at INSTALL time (the only deps mode since rcust P2b; Q3.2a pioneered it here): @@ -28,7 +28,7 @@ DEPS = ["keycloak"] # `abra app deploy`, and tests/lasuite-drive/install_steps.sh writes the OIDC env + client secret # into the .env that one deploy reads. No post-deploy reconverge (the flaky 12-service collabora # WOPI race is structurally gone). The post-deploy MinIO bucket one-shot lives in ops.py -# pre_install (the former setup_custom_tests.sh, deleted in P2b). +# pre_install (the former post-deploy setup hook, moved here in P2b). def READY_PROBE(ctx): diff --git a/tests/lasuite-meet/PARITY.md b/tests/lasuite-meet/PARITY.md index 622aa97..a11e24b 100644 --- a/tests/lasuite-meet/PARITY.md +++ b/tests/lasuite-meet/PARITY.md @@ -6,9 +6,9 @@ Reference corpus: `references/recipe-maintainer/recipe-info/lasuite-meet/tests/` | recipe-maintainer test | cc-ci test | what's verified (same thing) | |---|---|---| -| `health_check.py` | `tests/lasuite-meet/functional/test_health_check.py::test_lasuite_meet_returns_200` | HTTP 200/301/302 from `/` (SPA shell served). | -| `oidc_login.py` | `tests/lasuite-meet/functional/test_oidc_with_keycloak.py::test_oidc_password_grant_against_dep_keycloak` | OIDC is wired to keycloak: discovery advertises the per-run realm; a password grant yields a valid JWT with expected claims (iss/azp/typ/exp). Meet is OIDC-REQUIRED; OIDC wired at install (install_steps.sh, OIDC_AT_INSTALL). | -| `meeting_flow.py` | `tests/lasuite-meet/functional/test_meeting_flow.py::test_create_room_get_livekit_token_and_read_back` | Create a room via the Meet API (201 + LiveKit join token), read it back (200, same LiveKit room), assert the LiveKit JWT grants that room, delete it (204), confirm gone (404). | +| `health_check.py` | `tests/lasuite-meet/custom/test_health_check.py::test_lasuite_meet_returns_200` | HTTP 200/301/302 from `/` (SPA shell served). | +| `oidc_login.py` | `tests/lasuite-meet/custom/test_oidc_with_keycloak.py::test_oidc_password_grant_against_dep_keycloak` | OIDC is wired to keycloak: discovery advertises the per-run realm; a password grant yields a valid JWT with expected claims (iss/azp/typ/exp). Meet is OIDC-REQUIRED; OIDC wired at install (install_steps.sh, OIDC_AT_INSTALL). | +| `meeting_flow.py` | `tests/lasuite-meet/custom/test_meeting_flow.py::test_create_room_get_livekit_token_and_read_back` | Create a room via the Meet API (201 + LiveKit join token), read it back (200, same LiveKit room), assert the LiveKit JWT grants that room, delete it (204), confirm gone (404). | ## Recipe-specific functional tests (P3, ≥2 beyond bare parity) 1. **`test_meeting_flow.py`** — §4.3 create-an-object + read-it-back: create a room → GET it back → diff --git a/tests/lasuite-meet/functional/test_health_check.py b/tests/lasuite-meet/custom/test_health_check.py similarity index 100% rename from tests/lasuite-meet/functional/test_health_check.py rename to tests/lasuite-meet/custom/test_health_check.py diff --git a/tests/lasuite-meet/functional/test_meeting_flow.py b/tests/lasuite-meet/custom/test_meeting_flow.py similarity index 100% rename from tests/lasuite-meet/functional/test_meeting_flow.py rename to tests/lasuite-meet/custom/test_meeting_flow.py diff --git a/tests/lasuite-meet/functional/test_oidc_with_keycloak.py b/tests/lasuite-meet/custom/test_oidc_with_keycloak.py similarity index 97% rename from tests/lasuite-meet/functional/test_oidc_with_keycloak.py rename to tests/lasuite-meet/custom/test_oidc_with_keycloak.py index bac033d..27e741b 100644 --- a/tests/lasuite-meet/functional/test_oidc_with_keycloak.py +++ b/tests/lasuite-meet/custom/test_oidc_with_keycloak.py @@ -12,7 +12,7 @@ Mirrors the proven lasuite-docs SSO model: `deps-not-ready` reason — and (per F2-11) the orchestrator then fails the run rather than going green on a skipped SSO test. -SOURCE: adapted from tests/lasuite-docs/functional/test_oidc_with_keycloak.py (Q2.4 acceptance). +SOURCE: adapted from tests/lasuite-docs/custom/test_oidc_with_keycloak.py (Q2.4 acceptance). """ from __future__ import annotations diff --git a/tests/mailu/PARITY.md b/tests/mailu/PARITY.md index 63de667..f816442 100644 --- a/tests/mailu/PARITY.md +++ b/tests/mailu/PARITY.md @@ -16,10 +16,10 @@ email stack: nginx front + admin + postfix/smtp + dovecot/imap + rspamd/antispam (cc-ci-run) tests reach SMTP/IMAP at 127.0.0.1. ## Recipe-specific functional tests (P3 — ≥2) -1. `functional/test_mailbox.py` — §4.3 create-an-object + read-back: create a mailbox via the admin +1. `custom/test_mailbox.py` — §4.3 create-an-object + read-back: create a mailbox via the admin container's `flask mailu user` CLI, then read it back from `flask mailu config-export --json` and assert the address is present (admin-DB provisioning round-trip). -2. `functional/test_mail_flow.py` — the characteristic end-to-end mail flow: INJECT a uniquely-marked +2. `custom/test_mail_flow.py` — the characteristic end-to-end mail flow: INJECT a uniquely-marked message to the mailbox via the postfix container's local `sendmail` (locally-originated → not greylisted), then VERIFY delivery+storage via dovecot's `doveadm search` in the imap container — a real postfix → rspamd → dovecot deliver/store/fetch round-trip. We use the in-container mail diff --git a/tests/mailu/functional/_mailu.py b/tests/mailu/custom/_mailu.py similarity index 100% rename from tests/mailu/functional/_mailu.py rename to tests/mailu/custom/_mailu.py diff --git a/tests/mailu/functional/test_health_check.py b/tests/mailu/custom/test_health_check.py similarity index 100% rename from tests/mailu/functional/test_health_check.py rename to tests/mailu/custom/test_health_check.py diff --git a/tests/mailu/functional/test_mail_flow.py b/tests/mailu/custom/test_mail_flow.py similarity index 100% rename from tests/mailu/functional/test_mail_flow.py rename to tests/mailu/custom/test_mail_flow.py diff --git a/tests/mailu/functional/test_mailbox.py b/tests/mailu/custom/test_mailbox.py similarity index 100% rename from tests/mailu/functional/test_mailbox.py rename to tests/mailu/custom/test_mailbox.py diff --git a/tests/mailu/ops.py b/tests/mailu/ops.py index 75bb8d0..a7b9987 100644 --- a/tests/mailu/ops.py +++ b/tests/mailu/ops.py @@ -11,7 +11,7 @@ import time sys.path.insert(0, os.path.join(os.path.dirname(__file__), "..", "..", "runner")) from harness import lifecycle # noqa: E402 -sys.path.insert(0, os.path.join(os.path.dirname(__file__), "functional")) +sys.path.insert(0, os.path.join(os.path.dirname(__file__), "custom")) import _mailu # noqa: E402 _CI_LOCALPART = "citest" diff --git a/tests/mailu/test_backup.py b/tests/mailu/test_backup.py index e716044..7afe756 100644 --- a/tests/mailu/test_backup.py +++ b/tests/mailu/test_backup.py @@ -10,7 +10,7 @@ from __future__ import annotations import os import sys -sys.path.insert(0, os.path.join(os.path.dirname(__file__), "functional")) +sys.path.insert(0, os.path.join(os.path.dirname(__file__), "custom")) import _mailu # noqa: E402 sys.path.insert(0, os.path.join(os.path.dirname(__file__), "..", "..", "runner")) diff --git a/tests/mailu/test_restore.py b/tests/mailu/test_restore.py index ccab765..452710a 100644 --- a/tests/mailu/test_restore.py +++ b/tests/mailu/test_restore.py @@ -10,7 +10,7 @@ from __future__ import annotations import os import sys -sys.path.insert(0, os.path.join(os.path.dirname(__file__), "functional")) +sys.path.insert(0, os.path.join(os.path.dirname(__file__), "custom")) import _mailu # noqa: E402 sys.path.insert(0, os.path.join(os.path.dirname(__file__), "..", "..", "runner")) diff --git a/tests/matrix-synapse/PARITY.md b/tests/matrix-synapse/PARITY.md index 534735b..af5c59b 100644 --- a/tests/matrix-synapse/PARITY.md +++ b/tests/matrix-synapse/PARITY.md @@ -8,7 +8,7 @@ messages, federation) in Python tests adapted to the ephemeral per-run-deploy mo | recipe-maintainer file | cc-ci file | what's verified | status | |---|---|---|---| -| (no health_check.py in the recipe-maintainer corpus) | `tests/matrix-synapse/functional/test_health_check.py` | HTTP 200 + JSON document from `/_matrix/client/versions` (the synapse client API). | **Phase-2 health_check** (aligned with the parity-port convention; the corpus has no health_check.py to port from). | +| (no health_check.py in the recipe-maintainer corpus) | `tests/matrix-synapse/custom/test_health_check.py` | HTTP 200 + JSON document from `/_matrix/client/versions` (the synapse client API). | **Phase-2 health_check** (aligned with the parity-port convention; the corpus has no health_check.py to port from). | | `recipe-info/matrix-synapse/tests/compress_state.sh` | (deferred to Q4 follow-up — synapse_auto_compressor + state-group bloat) | The original creates state groups WITHOUT edges (full snapshots — Synapse's bloat pattern), runs the synapse_auto_compressor, asserts row counts drop. Requires per-run admin user pre-seeded + a long-running synapse + access to the synapse_auto_compressor binary. | **deferred** (operational complexity; needs custom install_steps.sh + admin user pre-seeding) | | `recipe-info/matrix-synapse/tests/test_complexity_limit.sh` | (deferred to Q4 follow-up — rate-limit behaviour) | Exercises Synapse's complexity-limit rejection of huge events. | **deferred** (load-test class; needs many-event setup) | | `recipe-info/matrix-synapse/tests/test_purge.sh` | (deferred to Q4 follow-up — admin purge commands) | Tests the abra.sh `db purge_history`, `db purge_room` etc. helpers. Operational tests against the recipe's helper shell wrappers. | **deferred** (recipe-helper-script tests, not synapse-behavior tests; orthogonal to Phase-2 P3) | @@ -28,8 +28,8 @@ Three specific tests landed (beyond parity health_check): | cc-ci file | what's verified | rationale | |---|---|---| -| `tests/matrix-synapse/functional/test_federation_version.py` | GET `/_matrix/federation/v1/version` → 200, JSON with `server.name == "Synapse"`, non-empty `server.version`. | Plan §4.3 prescribed. Federation discovery endpoint — the recipe's "is this a real Synapse, federation-ready" surface. Non-vacuous: a Dendrite or misconfigured federation subsystem fails. | -| `tests/matrix-synapse/functional/test_register_and_message.py` | **Plan §4.3 prescribed create-and-read-back.** Reads the abra-generated `registration` shared secret from the synapse container; registers two users (alice + bob) via `/_synapse/admin/v1/register` (HMAC-SHA1 nonce flow); both login via `/_matrix/client/v3/login`; alice creates a private_chat room; invites bob; bob joins; alice sends a uniquely-marked m.room.message; bob reads the room's messages and finds the marker. | The canonical Matrix create-and-read-back, exercising registration + login + room create/invite/join + send/receive across the full client API. Non-vacuous: each step fails at the level it's broken (admin API, login, room ops, send/receive); marker assertion confirms the message actually round-tripped across two users. | +| `tests/matrix-synapse/custom/test_federation_version.py` | GET `/_matrix/federation/v1/version` → 200, JSON with `server.name == "Synapse"`, non-empty `server.version`. | Plan §4.3 prescribed. Federation discovery endpoint — the recipe's "is this a real Synapse, federation-ready" surface. Non-vacuous: a Dendrite or misconfigured federation subsystem fails. | +| `tests/matrix-synapse/custom/test_register_and_message.py` | **Plan §4.3 prescribed create-and-read-back.** Reads the abra-generated `registration` shared secret from the synapse container; registers two users (alice + bob) via `/_synapse/admin/v1/register` (HMAC-SHA1 nonce flow); both login via `/_matrix/client/v3/login`; alice creates a private_chat room; invites bob; bob joins; alice sends a uniquely-marked m.room.message; bob reads the room's messages and finds the marker. | The canonical Matrix create-and-read-back, exercising registration + login + room create/invite/join + send/receive across the full client API. Non-vacuous: each step fails at the level it's broken (admin API, login, room ops, send/receive); marker assertion confirms the message actually round-tripped across two users. | Media upload/download deferred — would add a fourth specific test (`media_upload_roundtrip`) using `/_matrix/media/v3/upload` + `/_matrix/media/v3/download//`. Not in this diff --git a/tests/matrix-synapse/functional/test_federation_version.py b/tests/matrix-synapse/custom/test_federation_version.py similarity index 100% rename from tests/matrix-synapse/functional/test_federation_version.py rename to tests/matrix-synapse/custom/test_federation_version.py diff --git a/tests/matrix-synapse/functional/test_health_check.py b/tests/matrix-synapse/custom/test_health_check.py similarity index 100% rename from tests/matrix-synapse/functional/test_health_check.py rename to tests/matrix-synapse/custom/test_health_check.py diff --git a/tests/matrix-synapse/functional/test_register_and_message.py b/tests/matrix-synapse/custom/test_register_and_message.py similarity index 100% rename from tests/matrix-synapse/functional/test_register_and_message.py rename to tests/matrix-synapse/custom/test_register_and_message.py diff --git a/tests/mattermost-lts/PARITY.md b/tests/mattermost-lts/PARITY.md index 8993557..15164c5 100644 --- a/tests/mattermost-lts/PARITY.md +++ b/tests/mattermost-lts/PARITY.md @@ -12,14 +12,14 @@ tiers run by default (the Phase-1e invariant: no overlay ⇒ generic runs). The postgres in-compose (no external dep), so no dependency resolution is needed. ## P3 — Recipe-specific functional tests (≥2 separate characteristic tests) -1. `functional/test_create_message.py::test_create_message_roundtrip` — **§4.3 create-an-object + +1. `custom/test_create_message.py::test_create_message_roundtrip` — **§4.3 create-an-object + read-it-back**: first user (system admin) → login → create team → create channel → POST a unique marker message → GET it back by id → assert the text round-trips. -2. `functional/test_multiuser_message.py::test_second_user_reads_first_users_message` — the defining +2. `custom/test_multiuser_message.py::test_second_user_reads_first_users_message` — the defining **team-chat** behaviour (distinct code path: membership + ACL + cross-user delivery): user_a posts a unique marker; a SECOND user (created via admin API, added to team+channel) logs in with its own session and GETs the channel posts → asserts it sees user_a's message. Not a self read-back. -- `functional/test_health_check.py` — `test_root_serves` (`/` 200/302) + `test_system_ping_ok` +- `custom/test_health_check.py` — `test_root_serves` (`/` 200/302) + `test_system_ping_ok` (`/api/v4/system/ping` → `{"status":"OK"}`, API liveness). Supporting health/liveness, not counted toward the P3 ≥2 floor. diff --git a/tests/mattermost-lts/functional/_mm.py b/tests/mattermost-lts/custom/_mm.py similarity index 100% rename from tests/mattermost-lts/functional/_mm.py rename to tests/mattermost-lts/custom/_mm.py diff --git a/tests/mattermost-lts/functional/test_create_message.py b/tests/mattermost-lts/custom/test_create_message.py similarity index 100% rename from tests/mattermost-lts/functional/test_create_message.py rename to tests/mattermost-lts/custom/test_create_message.py diff --git a/tests/mattermost-lts/functional/test_health_check.py b/tests/mattermost-lts/custom/test_health_check.py similarity index 100% rename from tests/mattermost-lts/functional/test_health_check.py rename to tests/mattermost-lts/custom/test_health_check.py diff --git a/tests/mattermost-lts/functional/test_multiuser_message.py b/tests/mattermost-lts/custom/test_multiuser_message.py similarity index 100% rename from tests/mattermost-lts/functional/test_multiuser_message.py rename to tests/mattermost-lts/custom/test_multiuser_message.py diff --git a/tests/mumble/PARITY.md b/tests/mumble/PARITY.md index bcf9554..c53248e 100644 --- a/tests/mumble/PARITY.md +++ b/tests/mumble/PARITY.md @@ -12,9 +12,9 @@ publishes 64738 on the cc-ci host so the on-host (cc-ci-run) protocol tests conn | recipe-maintainer test (`recipe-info/mumble/tests/`) | what it verifies | cc-ci test | same thing? | |---|---|---|---| -| `health_check.py` | mumble server listening on TCP 64738 | `functional/test_tcp_health.py` | yes — TCP connect to 64738 (host-published) | -| `mumble_connect.py` | full TLS protocol handshake: TLS connect, server Version, auth accepted (no Reject), channel list present, ServerSync handshake completes, welcome text | `functional/test_protocol_handshake.py` (+ `functional/_mumble_proto.py`, adapted from the corpus's stdlib protobuf/protocol code) | yes — same handshake; asserts tls_connect + version + auth_accepted + channel presence + ServerSync | -| `web_client.py` | mumble-web client reachable over HTTPS, HTTP 200, page contains `Mumble` + `config.js`, valid HTML | `functional/test_web_client.py` | yes — same 200 + body markers (`Mumble`, `config.js`, ``) | +| `health_check.py` | mumble server listening on TCP 64738 | `custom/test_tcp_health.py` | yes — TCP connect to 64738 (host-published) | +| `mumble_connect.py` | full TLS protocol handshake: TLS connect, server Version, auth accepted (no Reject), channel list present, ServerSync handshake completes, welcome text | `custom/test_protocol_handshake.py` (+ `custom/_mumble_proto.py`, adapted from the corpus's stdlib protobuf/protocol code) | yes — same handshake; asserts tls_connect + version + auth_accepted + channel presence + ServerSync | +| `web_client.py` | mumble-web client reachable over HTTPS, HTTP 200, page contains `Mumble` + `config.js`, valid HTML | `custom/test_web_client.py` | yes — same 200 + body markers (`Mumble`, `config.js`, ``) | No recipe-maintainer mumble test is omitted — all three are ported. No `DECISIONS.md` non-port entry is needed for mumble. @@ -27,10 +27,10 @@ prove our deploy-time configuration propagated into the running murmur server an delivered over the real protocol (version-independent — they assert OUR configured markers, not hard-coded upstream values): -1. `functional/test_welcome_text_roundtrip.py` — deploys with a unique `WELCOME_TEXT` marker +1. `custom/test_welcome_text_roundtrip.py` — deploys with a unique `WELCOME_TEXT` marker (`recipe_meta.EXTRA_ENV` → `MUMBLE_CONFIG_WELCOMETEXT`); asserts that exact marker surfaces in the server's ServerSync `welcome_text` delivered to a connecting client. (create config → read back.) -2. `functional/test_server_config_limits.py` — deploys with a distinctive non-default `USERS=42` +2. `custom/test_server_config_limits.py` — deploys with a distinctive non-default `USERS=42` (max-users cap → `MUMBLE_CONFIG_USERS`); asserts the server's ServerConfig message reports `max_users == 42` (and a well-formed `allow_html`), proving the recipe wires deploy-time server-capacity policy into the running server. diff --git a/tests/mumble/functional/_mumble_proto.py b/tests/mumble/custom/_mumble_proto.py similarity index 100% rename from tests/mumble/functional/_mumble_proto.py rename to tests/mumble/custom/_mumble_proto.py diff --git a/tests/mumble/functional/test_protocol_handshake.py b/tests/mumble/custom/test_protocol_handshake.py similarity index 100% rename from tests/mumble/functional/test_protocol_handshake.py rename to tests/mumble/custom/test_protocol_handshake.py diff --git a/tests/mumble/functional/test_server_config_limits.py b/tests/mumble/custom/test_server_config_limits.py similarity index 100% rename from tests/mumble/functional/test_server_config_limits.py rename to tests/mumble/custom/test_server_config_limits.py diff --git a/tests/mumble/functional/test_tcp_health.py b/tests/mumble/custom/test_tcp_health.py similarity index 100% rename from tests/mumble/functional/test_tcp_health.py rename to tests/mumble/custom/test_tcp_health.py diff --git a/tests/mumble/functional/test_web_client.py b/tests/mumble/custom/test_web_client.py similarity index 100% rename from tests/mumble/functional/test_web_client.py rename to tests/mumble/custom/test_web_client.py diff --git a/tests/mumble/functional/test_welcome_text_roundtrip.py b/tests/mumble/custom/test_welcome_text_roundtrip.py similarity index 100% rename from tests/mumble/functional/test_welcome_text_roundtrip.py rename to tests/mumble/custom/test_welcome_text_roundtrip.py diff --git a/tests/n8n/PARITY.md b/tests/n8n/PARITY.md index b5e4084..4ae005f 100644 --- a/tests/n8n/PARITY.md +++ b/tests/n8n/PARITY.md @@ -1,13 +1,13 @@ # Parity — n8n Phase-2 P2 mapping table: every `references/recipe-maintainer/recipe-info/n8n/tests/*.py` has a -comparable cc-ci test under `tests/n8n/functional/`, asserting the **same thing** (not a renamed +comparable cc-ci test under `tests/n8n/custom/`, asserting the **same thing** (not a renamed file). The Adversary cold-verifies parity by reading the source `recipe-info/` and the cc-ci file side-by-side. | recipe-maintainer file | cc-ci file | what's verified | status | |---|---|---|---| -| `recipe-info/n8n/tests/health_check.py` | `tests/n8n/functional/test_health_check.py` | The app is reachable over HTTPS and returns a successful response (the original asserted HTTP 200 against a persistent `n8n.` host). The cc-ci port preserves the assertion shape — HTTP 200 from the served root — and adapts to the ephemeral per-run domain via the `live_app` fixture. | **ported** | +| `recipe-info/n8n/tests/health_check.py` | `tests/n8n/custom/test_health_check.py` | The app is reachable over HTTPS and returns a successful response (the original asserted HTTP 200 against a persistent `n8n.` host). The cc-ci port preserves the assertion shape — HTTP 200 from the served root — and adapts to the ephemeral per-run domain via the `live_app` fixture. | **ported** | ## Recipe-specific tests (Phase-2 P3 §4.3 floor: "create-an-object + read-it-back, and one more") @@ -16,9 +16,9 @@ directly: "create a workflow via API, execute it, assert the result." So: | cc-ci file | what's verified | rationale | |---|---|---| -| `tests/n8n/functional/test_workflow_roundtrip.py` | Owner setup via `POST /rest/owner/setup` with a per-run-generated email + password (class-B run-scoped secret, plan §4.4-B); then `POST /rest/workflows` creates a Manual-Trigger workflow with a unique name; then `GET /rest/workflows/` reads it back; asserts the returned id matches, name matches, nodes payload preserved (type/name of the one node). | **Plan §4.3 prescribed test** — create-an-object + read-it-back, exercising n8n's persistence + retrieval. Non-vacuous: a broken persistence layer would round-trip with wrong shape; a wedged engine that serves the SPA but rejects workflow POSTs fails at the create step. | -| `tests/n8n/functional/test_rest_settings.py` | Polls `/rest/settings` until response is **application/json** (rejects the "n8n is starting up" SPA placeholder); asserts known public-settings keys (`userManagement`, `defaultLocale`, `authCookie`) in the `data` envelope. | The editor SPA's primary API contract — proves bootstrap surface is intact. Distinct from `test_workflow_roundtrip.py` (which proves persistence); this proves the SPA can come up at all. | -| `tests/n8n/functional/test_login_state.py` | Polls `/rest/login` until response is **application/json**; asserts JSON dict/list shape — proves the user-management/auth subsystem initialized. | Auth subsystem readiness; distinct from settings (a broken auth backend would let settings return JSON but login would 5xx). | +| `tests/n8n/custom/test_workflow_roundtrip.py` | Owner setup via `POST /rest/owner/setup` with a per-run-generated email + password (class-B run-scoped secret, plan §4.4-B); then `POST /rest/workflows` creates a Manual-Trigger workflow with a unique name; then `GET /rest/workflows/` reads it back; asserts the returned id matches, name matches, nodes payload preserved (type/name of the one node). | **Plan §4.3 prescribed test** — create-an-object + read-it-back, exercising n8n's persistence + retrieval. Non-vacuous: a broken persistence layer would round-trip with wrong shape; a wedged engine that serves the SPA but rejects workflow POSTs fails at the create step. | +| `tests/n8n/custom/test_rest_settings.py` | Polls `/rest/settings` until response is **application/json** (rejects the "n8n is starting up" SPA placeholder); asserts known public-settings keys (`userManagement`, `defaultLocale`, `authCookie`) in the `data` envelope. | The editor SPA's primary API contract — proves bootstrap surface is intact. Distinct from `test_workflow_roundtrip.py` (which proves persistence); this proves the SPA can come up at all. | +| `tests/n8n/custom/test_login_state.py` | Polls `/rest/login` until response is **application/json**; asserts JSON dict/list shape — proves the user-management/auth subsystem initialized. | Auth subsystem readiness; distinct from settings (a broken auth backend would let settings return JSON but login would 5xx). | Three specific tests, exceeding the ≥2 floor — `test_workflow_roundtrip.py` is the plan §4.3 prescribed "create + read-back"; the other two are bootstrap-readiness assertions retained from diff --git a/tests/n8n/functional/test_health_check.py b/tests/n8n/custom/test_health_check.py similarity index 100% rename from tests/n8n/functional/test_health_check.py rename to tests/n8n/custom/test_health_check.py diff --git a/tests/n8n/functional/test_login_state.py b/tests/n8n/custom/test_login_state.py similarity index 100% rename from tests/n8n/functional/test_login_state.py rename to tests/n8n/custom/test_login_state.py diff --git a/tests/n8n/functional/test_rest_settings.py b/tests/n8n/custom/test_rest_settings.py similarity index 100% rename from tests/n8n/functional/test_rest_settings.py rename to tests/n8n/custom/test_rest_settings.py diff --git a/tests/n8n/functional/test_workflow_roundtrip.py b/tests/n8n/custom/test_workflow_roundtrip.py similarity index 100% rename from tests/n8n/functional/test_workflow_roundtrip.py rename to tests/n8n/custom/test_workflow_roundtrip.py diff --git a/tests/plausible/PARITY.md b/tests/plausible/PARITY.md index 4a3d92c..6f45d02 100644 --- a/tests/plausible/PARITY.md +++ b/tests/plausible/PARITY.md @@ -20,10 +20,10 @@ a true readiness gate. `DEPLOY_TIMEOUT` / `HTTP_TIMEOUT` are widened to 1200s to ClickHouse + migrations init. ## P3 — Recipe-specific functional tests -- `functional/test_health_check.py` +- `custom/test_health_check.py` - `test_plausible_root_serves` — GET `/api/health` → 200, proving ClickHouse + postgres + the sites_cache are all up (plausible's self-reported backend readiness; not a Traefik fallback). -- `functional/test_event_tracking.py` — **§4.3 prescribed "track a test event, query it back"**, the +- `custom/test_event_tracking.py` — **§4.3 prescribed "track a test event, query it back"**, the app's primary object. Both tests register a site row in the metadata postgres (plausible's `sites_cache` drops events for unregistered domains — empirically confirmed), POST to the public `/api/event` ingestion endpoint with a browser User-Agent (plausible drops bot/library UAs), then diff --git a/tests/plausible/functional/test_event_tracking.py b/tests/plausible/custom/test_event_tracking.py similarity index 100% rename from tests/plausible/functional/test_event_tracking.py rename to tests/plausible/custom/test_event_tracking.py diff --git a/tests/plausible/functional/test_health_check.py b/tests/plausible/custom/test_health_check.py similarity index 100% rename from tests/plausible/functional/test_health_check.py rename to tests/plausible/custom/test_health_check.py diff --git a/tests/unit/test_discovery.py b/tests/unit/test_discovery.py index e170f86..b2baed0 100644 --- a/tests/unit/test_discovery.py +++ b/tests/unit/test_discovery.py @@ -72,17 +72,16 @@ def test_repo_local_wins_when_approved(tmp_path): def test_custom_tests_repo_local_gated(tmp_path, monkeypatch): # custom test_*.py from repo-local only count for approved recipes (HC2); placement rule - # (rcust P4): custom tests live under functional/ (or playwright/) — top-level files are - # lifecycle overlays only, so the repo-local custom here sits in functional/. + # (cfold): custom/ is canonical, while functional/ and playwright/ remain deprecated aliases. # Use a synthetic recipe name + monkeypatched cc_ci_dir so this is independent of what # tests// ships (F2-1). fake_recipe = "ccci-hc2-fixture" monkeypatch.setattr(discovery, "cc_ci_dir", lambda r: str(tmp_path / "cc-ci" / r)) (tmp_path / "cc-ci" / fake_recipe).mkdir(parents=True) rl = tmp_path / "repo" - (rl / "functional").mkdir(parents=True) - (rl / "functional" / "test_sso.py").write_text("# repo-local custom\n") - (rl / "functional" / "test_install.py").write_text("# lifecycle name -> excluded from custom\n") + (rl / "custom").mkdir(parents=True) + (rl / "custom" / "test_sso.py").write_text("# repo-local custom\n") + (rl / "custom" / "test_install.py").write_text("# lifecycle name -> excluded from custom\n") _approve(tmp_path) # not approved -> repo-local custom ignored assert discovery.custom_tests(fake_recipe, str(rl)) == [] @@ -94,6 +93,25 @@ def test_custom_tests_repo_local_gated(tmp_path, monkeypatch): assert all(os.path.basename(p) != "test_install.py" for _, p in customs) +def test_custom_tests_prefers_custom_and_warns_on_deprecated_aliases(tmp_path, monkeypatch, capsys): + fake_recipe = "ccci-cfold-fixture" + fake_dir = tmp_path / "tests" / fake_recipe + (fake_dir / "custom").mkdir(parents=True) + (fake_dir / "functional").mkdir() + (fake_dir / "playwright").mkdir() + (fake_dir / "custom" / "test_a.py").write_text("# canonical\n") + (fake_dir / "functional" / "test_b.py").write_text("# deprecated alias\n") + (fake_dir / "playwright" / "test_c.py").write_text("# deprecated alias\n") + monkeypatch.setattr(discovery, "cc_ci_dir", lambda r: str(tmp_path / "tests" / r)) + + customs = discovery.custom_tests(fake_recipe, None) + + assert [os.path.basename(path) for _, path in customs] == ["test_a.py", "test_b.py", "test_c.py"] + err = capsys.readouterr().err + assert "deprecated folder 'functional/'" in err + assert "deprecated folder 'playwright/'" in err + + def test_install_steps_repo_local_gated(tmp_path): rl = tmp_path / "repo" rl.mkdir() diff --git a/tests/unit/test_discovery_phase2.py b/tests/unit/test_discovery_phase2.py index 882b4e6..e1803cd 100644 --- a/tests/unit/test_discovery_phase2.py +++ b/tests/unit/test_discovery_phase2.py @@ -1,8 +1,8 @@ """Unit tests for Phase-2 discovery additions (plan §4.1). -Proves the `custom_tests` discovery covers exactly the per-recipe `functional/` + `playwright/` -subdirs as well as the top-level dir, while still excluding lifecycle `test_.py` names and -honouring the HC2 repo-local approval gate. +Proves the `custom_tests` discovery covers exactly the per-recipe `custom/` dir, still honors the +deprecated `functional/` + `playwright/` aliases, excludes top-level lifecycle `test_.py` +names, and honors the HC2 repo-local approval gate. Run with: `cc-ci-run -m pytest tests/unit`. Located under tests/unit/ so the orchestrator never picks these up as overlays/custom tests. @@ -27,29 +27,24 @@ def teardown_function(): os.environ.pop("CCCI_REPO_LOCAL_APPROVED_FILE", None) -def test_custom_tests_placement_rule_functional_playwright_only(tmp_path, monkeypatch): - """Placement rule (rcust P4): custom tests are discovered ONLY under functional/ + - playwright/. A top-level non-lifecycle test_*.py is NOT discovered (top level is reserved - for lifecycle overlays); lifecycle names inside the subdirs stay excluded (defensive).""" - # Point cc-ci's per-recipe dir at a fake recipe in tmp_path +def test_custom_tests_placement_rule_custom_only(tmp_path, monkeypatch): + """Placement rule (cfold): custom tests are discovered under custom/. A top-level + non-lifecycle test_*.py is NOT discovered (top level is reserved for lifecycle overlays); + lifecycle names inside custom/ stay excluded (defensive).""" fake_recipe = "ccci-phase2-fixture" fake_dir = tmp_path / "tests" / fake_recipe - (fake_dir / "functional").mkdir(parents=True) - (fake_dir / "playwright").mkdir() + (fake_dir / "custom").mkdir(parents=True) (fake_dir / "test_sso_smoke.py").write_text("# top-level — NOT discovered since P4\n") - (fake_dir / "functional" / "test_health_check.py").write_text("# parity port\n") - (fake_dir / "functional" / "test_content_roundtrip.py").write_text("# recipe-specific\n") - (fake_dir / "playwright" / "test_login_flow.py").write_text("# UI flow\n") - # lifecycle name in functional/ should be ignored (defensive) - (fake_dir / "functional" / "test_install.py").write_text("# misfiled lifecycle name\n") + (fake_dir / "custom" / "test_health_check.py").write_text("# parity port\n") + (fake_dir / "custom" / "test_content_roundtrip.py").write_text("# recipe-specific\n") + (fake_dir / "custom" / "test_login_flow.py").write_text("# UI flow\n") + (fake_dir / "custom" / "test_install.py").write_text("# misfiled lifecycle name\n") - # Patch the cc-ci dir resolver to point at our fixture monkeypatch.setattr(discovery, "cc_ci_dir", lambda r: str(tmp_path / "tests" / r)) customs = discovery.custom_tests(fake_recipe, None) names = sorted((src, os.path.basename(p)) for src, p in customs) - # functional/ + playwright/ discovered; top-level custom + lifecycle name are NOT assert ("cc-ci", "test_health_check.py") in names assert ("cc-ci", "test_content_roundtrip.py") in names assert ("cc-ci", "test_login_flow.py") in names @@ -58,20 +53,20 @@ def test_custom_tests_placement_rule_functional_playwright_only(tmp_path, monkey def test_custom_tests_repo_local_subdirs_gated(tmp_path, monkeypatch): - """HC2 gate still applies to functional/playwright subdirs under repo-local: not approved -> the - repo-local subdir contents are ignored even if they exist.""" + """HC2 gate still applies to custom/ under repo-local: not approved -> the repo-local + subdir contents are ignored even if they exist.""" fake_recipe = "ccci-phase2-fixture" monkeypatch.setattr(discovery, "cc_ci_dir", lambda r: str(tmp_path / "cc-ci" / r)) (tmp_path / "cc-ci" / fake_recipe).mkdir(parents=True) rl = tmp_path / "repo" - (rl / "functional").mkdir(parents=True) - (rl / "functional" / "test_repo_local_specific.py").write_text("# repo-local custom\n") + (rl / "custom").mkdir(parents=True) + (rl / "custom" / "test_repo_local_specific.py").write_text("# repo-local custom\n") - _approve(tmp_path) # empty allowlist → default-deny + _approve(tmp_path) assert discovery.custom_tests(fake_recipe, str(rl)) == [] - _approve(tmp_path, fake_recipe) # approved → repo-local subdir honored + _approve(tmp_path, fake_recipe) customs = discovery.custom_tests(fake_recipe, str(rl)) names = {(src, os.path.basename(p)) for src, p in customs} assert ("repo-local", "test_repo_local_specific.py") in names diff --git a/tests/unit/test_manifest.py b/tests/unit/test_manifest.py index e415c5e..dc6e241 100644 --- a/tests/unit/test_manifest.py +++ b/tests/unit/test_manifest.py @@ -24,13 +24,12 @@ def _mk_synthetic(tmp_path, monkeypatch, approved=True): """A synthetic recipe dir exercising EVERY manifest surface, plus a repo-local tests dir. cc-ci side: meta (2 data keys + 1 hook key non-default), ops.py (2 pre-ops), install_steps.sh, - compose.ccci.yml, test_backup.py overlay, 2 functional + 1 playwright custom tests. - repo-local side: test_restore.py overlay + 1 functional custom test (visible iff approved, HC2). + compose.ccci.yml, test_backup.py overlay, 3 custom tests. + repo-local side: test_restore.py overlay + 1 custom test (visible iff approved, HC2). """ ccci_root = tmp_path / "cc-ci-tests" d = ccci_root / RECIPE - (d / "functional").mkdir(parents=True) - (d / "playwright").mkdir() + (d / "custom").mkdir(parents=True) (d / "recipe_meta.py").write_text( "HTTP_TIMEOUT = 600\n" "DEPS = ['keycloak']\n" @@ -41,17 +40,17 @@ def _mk_synthetic(tmp_path, monkeypatch, approved=True): (d / "install_steps.sh").write_text("#!/usr/bin/env bash\n") (d / "compose.ccci.yml").write_text("version: '3.8'\n") (d / "test_backup.py").write_text("# lifecycle overlay\n") - (d / "functional" / "test_a.py").write_text("# custom\n") - (d / "functional" / "test_b.py").write_text("# custom\n") - (d / "playwright" / "test_ui.py").write_text("# custom\n") + (d / "custom" / "test_a.py").write_text("# custom\n") + (d / "custom" / "test_b.py").write_text("# custom\n") + (d / "custom" / "test_ui.py").write_text("# custom\n") rl = tmp_path / "repo-local" - (rl / "functional").mkdir(parents=True) - (rl / "functional" / "test_c.py").write_text("# repo-local custom\n") + (rl / "custom").mkdir(parents=True) + (rl / "custom" / "test_c.py").write_text("# repo-local custom\n") (rl / "test_restore.py").write_text("# repo-local lifecycle overlay\n") monkeypatch.setattr(discovery, "cc_ci_dir", lambda r: str(ccci_root / r)) - monkeypatch.setattr(meta_mod, "TESTS_DIR", str(ccci_root)) # compose.ccci.yml discovery + monkeypatch.setattr(meta_mod, "TESTS_DIR", str(ccci_root)) approved_file = tmp_path / "approved.txt" approved_file.write_text(f"{RECIPE}\n" if approved else "") monkeypatch.setenv("CCCI_REPO_LOCAL_APPROVED_FILE", str(approved_file)) @@ -61,7 +60,6 @@ def _mk_synthetic(tmp_path, monkeypatch, approved=True): def test_manifest_complete(tmp_path, monkeypatch): - # Every surface the synthetic recipe customizes appears — nothing silently dropped (R4). meta, rl = _mk_synthetic(tmp_path, monkeypatch) m = manifest.build(RECIPE, meta, rl) assert m["meta_non_default"] == { @@ -76,8 +74,8 @@ def test_manifest_complete(tmp_path, monkeypatch): } assert m["overlays"] == {"backup": "cc-ci", "restore": "repo-local"} assert m["custom_tests"] == { - "cc-ci": {"functional": 2, "playwright": 1}, - "repo-local": {"functional": 1}, + "cc-ci": {"custom": 3}, + "repo-local": {"custom": 1}, } assert m["env_overrides"] == [] @@ -87,11 +85,10 @@ def test_manifest_deterministic_and_serializable(tmp_path, monkeypatch): a = manifest.build(RECIPE, meta, rl) b = manifest.build(RECIPE, meta, rl) assert json.dumps(a, sort_keys=True) == json.dumps(b, sort_keys=True) - assert json.loads(json.dumps(a)) == a # round-trips: no callables/tuples leak through + assert json.loads(json.dumps(a)) == a def test_manifest_zero_config_floor(tmp_path, monkeypatch): - # A recipe with NO customization at all -> every section empty, render says so explicitly. ccci_root = tmp_path / "cc-ci-tests" (ccci_root / RECIPE).mkdir(parents=True) monkeypatch.setattr(discovery, "cc_ci_dir", lambda r: str(ccci_root / r)) @@ -112,31 +109,25 @@ def test_manifest_zero_config_floor(tmp_path, monkeypatch): def test_manifest_repo_local_hc2_gate(tmp_path, monkeypatch): - # Unapproved recipe -> repo-local overlay + custom tests INVISIBLE (same default-deny as the - # discovery they ride on; the manifest must not advertise code the run will not execute). meta, rl = _mk_synthetic(tmp_path, monkeypatch, approved=False) m = manifest.build(RECIPE, meta, rl) - assert m["overlays"] == {"backup": "cc-ci"} # repo-local test_restore.py gone + assert m["overlays"] == {"backup": "cc-ci"} assert "repo-local" not in m["custom_tests"] def test_manifest_env_overrides_and_ci_flag(tmp_path, monkeypatch): meta, rl = _mk_synthetic(tmp_path, monkeypatch) monkeypatch.setenv("CCCI_SKIP_GENERIC_BACKUP", "1") - monkeypatch.setenv("CCCI_SKIP_GENERIC_UPGRADE", "0") # falsy -> not an active override + monkeypatch.setenv("CCCI_SKIP_GENERIC_UPGRADE", "0") m = manifest.build(RECIPE, meta, rl) assert m["env_overrides"] == ["CCCI_SKIP_GENERIC_BACKUP"] monkeypatch.delenv("DRONE", raising=False) - assert "!!" not in manifest.render(RECIPE, m) # local dev: no CI warning - monkeypatch.setenv("DRONE", "true") # riding a CI run -> loud flag (P2c) + assert "!!" not in manifest.render(RECIPE, m) + monkeypatch.setenv("DRONE", "true") assert "!! dev-only override active in CI" in manifest.render(RECIPE, m) def test_manifest_redacts_sensitive_named_values(tmp_path, monkeypatch): - # Meta values are repo-public by construction, but the manifest lands on the dashboard: - # secret-NAMED entries (top-level or nested dict keys, e.g. plausible's - # EXTRA_ENV["SECRET_KEY_BASE"] dummy) render as '' — name shown, value masked. - # Non-sensitive names (incl. KEYCLOAK_* — 'KEY' matches only as a word segment) pass through. ccci_root = tmp_path / "cc-ci-tests" d = ccci_root / RECIPE d.mkdir(parents=True) @@ -159,7 +150,7 @@ def test_manifest_redacts_sensitive_named_values(tmp_path, monkeypatch): } out = manifest.render(RECIPE, m) assert "dummy-ci-constant" not in out and "also-dummy" not in out - assert "SECRET_KEY_BASE" in out # the key NAME stays visible + assert "SECRET_KEY_BASE" in out def test_render_lists_every_surface(tmp_path, monkeypatch): @@ -173,5 +164,5 @@ def test_render_lists_every_surface(tmp_path, monkeypatch): in lines ) assert "overlays: test_backup.py(cc-ci) test_restore.py(repo-local)" in lines - assert "custom tests: functional/=2 playwright/=1 (cc-ci) functional/=1 (repo-local)" in lines + assert "custom tests: custom/=3 (cc-ci) custom/=1 (repo-local)" in lines assert "env overrides: (none)" in lines diff --git a/tests/uptime-kuma/PARITY.md b/tests/uptime-kuma/PARITY.md index c9f4893..9d732d6 100644 --- a/tests/uptime-kuma/PARITY.md +++ b/tests/uptime-kuma/PARITY.md @@ -17,9 +17,9 @@ behaviors: | cc-ci file | what's verified | rationale | |---|---|---| -| `tests/uptime-kuma/functional/test_socketio_handshake.py` | GETs `/socket.io/?EIO=4&transport=polling` → 200 + Engine.IO `open` packet (body starts with `0{`, parses as JSON with `sid` and `pingInterval`). | Proves the **real-time backend is wired** through the nginx proxy. Non-vacuous: a wedged Socket.IO returns 404/502 here; a misrouted nginx returns 404. Only a correctly-wired uptime-kuma + Engine.IO listener completes the handshake. | -| `tests/uptime-kuma/functional/test_spa_branding.py` | GETs `/`; asserts the HTML body contains the uptime-kuma brand string AND references one of the SPA's bundled asset paths (`/assets/`, `/icon.svg`, `favicon`, `main.`). | Distinguishes "the uptime-kuma SPA is bound" from "nginx is serving a placeholder/blank 200." Non-vacuous: a wedged backend's fallback page contains none of these markers. | -| `tests/uptime-kuma/playwright/test_monitor_wizard.py` | Playwright browser test: completes the first-run setup wizard (admin create), creates an HTTP monitor targeting the app's own root URL, waits ≤90 s for the monitor to report **Up** with a real heartbeat timestamp, then creates a second monitor on a dead port and asserts **Down**. | §4.3 prescribed test — proves uptime-kuma's actual monitoring function (wizard + probe engine), not just that the SPA renders. Non-vacuous: Up requires a real outbound HTTP probe to succeed; Down requires the probe to detect connection-refused. Status is driven by Socket.IO heartbeat events from the server, not echoed from config. Green in drone build #460 (LEVEL 5). Phase kuma, 2026-06-11. | +| `tests/uptime-kuma/custom/test_socketio_handshake.py` | GETs `/socket.io/?EIO=4&transport=polling` → 200 + Engine.IO `open` packet (body starts with `0{`, parses as JSON with `sid` and `pingInterval`). | Proves the **real-time backend is wired** through the nginx proxy. Non-vacuous: a wedged Socket.IO returns 404/502 here; a misrouted nginx returns 404. Only a correctly-wired uptime-kuma + Engine.IO listener completes the handshake. | +| `tests/uptime-kuma/custom/test_spa_branding.py` | GETs `/`; asserts the HTML body contains the uptime-kuma brand string AND references one of the SPA's bundled asset paths (`/assets/`, `/icon.svg`, `favicon`, `main.`). | Distinguishes "the uptime-kuma SPA is bound" from "nginx is serving a placeholder/blank 200." Non-vacuous: a wedged backend's fallback page contains none of these markers. | +| `tests/uptime-kuma/custom/test_monitor_wizard.py` | Playwright browser test: completes the first-run setup wizard (admin create), creates an HTTP monitor targeting the app's own root URL, waits ≤90 s for the monitor to report **Up** with a real heartbeat timestamp, then creates a second monitor on a dead port and asserts **Down**. | §4.3 prescribed test — proves uptime-kuma's actual monitoring function (wizard + probe engine), not just that the SPA renders. Non-vacuous: Up requires a real outbound HTTP probe to succeed; Down requires the probe to detect connection-refused. Status is driven by Socket.IO heartbeat events from the server, not echoed from config. Green in drone build #460 (LEVEL 5). Phase kuma, 2026-06-11. | ## Backup data-integrity (P4) diff --git a/tests/uptime-kuma/functional/test_health_check.py b/tests/uptime-kuma/custom/test_health_check.py similarity index 100% rename from tests/uptime-kuma/functional/test_health_check.py rename to tests/uptime-kuma/custom/test_health_check.py diff --git a/tests/uptime-kuma/playwright/test_monitor_wizard.py b/tests/uptime-kuma/custom/test_monitor_wizard.py similarity index 100% rename from tests/uptime-kuma/playwright/test_monitor_wizard.py rename to tests/uptime-kuma/custom/test_monitor_wizard.py diff --git a/tests/uptime-kuma/functional/test_socketio_handshake.py b/tests/uptime-kuma/custom/test_socketio_handshake.py similarity index 100% rename from tests/uptime-kuma/functional/test_socketio_handshake.py rename to tests/uptime-kuma/custom/test_socketio_handshake.py diff --git a/tests/uptime-kuma/functional/test_spa_branding.py b/tests/uptime-kuma/custom/test_spa_branding.py similarity index 100% rename from tests/uptime-kuma/functional/test_spa_branding.py rename to tests/uptime-kuma/custom/test_spa_branding.py