feat(cfold): canonicalize custom test layout
Some checks failed
continuous-integration/drone/push Build is failing
Some checks failed
continuous-integration/drone/push Build is failing
This commit is contained in:
autonomic-bot
@ -22,7 +22,7 @@ secrets/ sops-encrypted infra secrets (cc-ci-secrets submodule)
|
||||
bridge/ !testme webhook listener source
|
||||
runner/ run_recipe_ci.py + shared pytest harness
|
||||
dashboard/ results overview generator
|
||||
tests/<recipe>/ per-recipe install/upgrade/backup tests + playwright/
|
||||
tests/<recipe>/ per-recipe install/upgrade/backup tests + custom/
|
||||
docs/ install, enroll-recipe, secrets, architecture, runbook, baseline
|
||||
```
|
||||
|
||||
|
||||
@ -22,12 +22,11 @@ tests/<recipe>/
|
||||
├── test_backup.py # optional backup overlay (runs ADDITIVELY alongside generic)
|
||||
├── test_restore.py # optional restore overlay (runs ADDITIVELY alongside generic)
|
||||
├── PARITY.md # Phase 2 P2: mapping table (recipe-maintainer tests → cc-ci tests)
|
||||
├── functional/ # Phase 2 P3: parity ports + ≥2 NEW recipe-specific tests
|
||||
│ ├── test_health_check.py # parity port of recipe-info/<recipe>/tests/health_check.py
|
||||
│ ├── test_<behavior>.py # ≥2 NEW recipe-specific functional tests
|
||||
│ └── …
|
||||
└── playwright/ # Phase 2 P6: browser flows where the app's core UX is a UI
|
||||
└── test_<flow>.py
|
||||
└── custom/ # custom tier: parity ports + recipe-specific tests + browser flows
|
||||
├── test_health_check.py # parity port of recipe-info/<recipe>/tests/health_check.py
|
||||
├── test_<behavior>.py # ≥2 NEW recipe-specific tests
|
||||
├── test_<flow>.py # browser/UI flows where relevant
|
||||
└── …
|
||||
```
|
||||
|
||||
**A recipe is testable with ZERO config:** with no overlay files, the **generic lifecycle suite**
|
||||
@ -68,18 +67,18 @@ ops themselves are orchestrator-owned (you never call them from an overlay). The
|
||||
Beyond the lifecycle overlays, each recipe carries (plan §4.1):
|
||||
|
||||
- **`PARITY.md`** — a mapping table from every `references/recipe-maintainer/recipe-info/<recipe>/
|
||||
tests/*.py` to a comparable cc-ci test under `tests/<recipe>/functional/`, asserting the
|
||||
tests/*.py` to a comparable cc-ci test under `tests/<recipe>/custom/`, asserting the
|
||||
*same thing* (not a renamed file). A deliberate non-port is documented in `DECISIONS.md` with
|
||||
a technical reason — never a silent omission.
|
||||
- **`functional/`** — parity-port tests + **≥2 NEW recipe-specific functional tests** that
|
||||
exercise the app's characteristic behavior (per plan §4.3 — e.g. "create-an-object +
|
||||
read-it-back, and one more that touches a distinctive feature"). Each parity-port file carries
|
||||
a `SOURCE = "recipe-info/<recipe>/tests/<file>"` comment near the top so audit is in-file.
|
||||
- **`playwright/`** — browser flows where the recipe's core UX is a UI (P6).
|
||||
- **`custom/`** — parity-port tests + **≥2 NEW recipe-specific tests** that exercise the app's
|
||||
characteristic behavior (per plan §4.3 — e.g. "create-an-object + read-it-back, and one more
|
||||
that touches a distinctive feature"). Browser/UI flows live in the same folder too. Each
|
||||
parity-port file carries a `SOURCE = "recipe-info/<recipe>/tests/<file>"` comment near the top
|
||||
so audit is in-file.
|
||||
|
||||
The orchestrator's **custom** tier discovers `test_*.py` in `tests/<recipe>/{functional,playwright}/`
|
||||
ONLY (the placement rule, via `runner/harness/discovery.custom_tests` — a top-level `test_*.py`
|
||||
is a lifecycle overlay and nothing else) and runs each as its own pytest against the same
|
||||
The orchestrator's **custom** tier discovers `test_*.py` in canonical `tests/<recipe>/custom/`
|
||||
(plus deprecated `functional/` / `playwright/` aliases during migration; discovery warns when it
|
||||
uses them) and runs each as its own pytest against the same
|
||||
`live_app` shared deployment. Lifecycle-named files (`test_install.py`/etc.) are **excluded**
|
||||
from the custom tier even inside those subdirs (safety net against double-running).
|
||||
|
||||
@ -176,7 +175,7 @@ shapes (proven on mumble, mailu, and the SSO-dependent suite):
|
||||
|
||||
**Non-HTTP protocol tests (mumble).** Reach a TCP service published `mode: host` (via a host-ports
|
||||
overlay) at `127.0.0.1:<port>` — cc-ci runs tests on-host (cc-ci-run). mumble ships a stdlib protocol
|
||||
client (`tests/mumble/functional/_mumble_proto.py`) doing the real TLS handshake → ServerSync; the
|
||||
client (`tests/mumble/custom/_mumble_proto.py`) doing the real TLS handshake → ServerSync; the
|
||||
recipe-specific tests assert channel presence and config round-trips (a deploy-set `WELCOME_TEXT`/
|
||||
`USERS` value surfaces over the protocol — version-independent, non-vacuous).
|
||||
|
||||
@ -244,7 +243,7 @@ tests/lasuite-docs/
|
||||
├── test_backup.py # lifecycle backup overlay (marker captured)
|
||||
├── test_restore.py # lifecycle restore overlay (marker restored to pre-mutation)
|
||||
├── PARITY.md # parity-port mapping (P2)
|
||||
└── functional/
|
||||
└── custom/
|
||||
├── test_health_check.py # parity port (SOURCE comment cites recipe-info file)
|
||||
├── test_auth_required.py # specific: /api/v1.0/users/me/ → 401 without auth
|
||||
└── test_oidc_with_keycloak.py # specific: full OIDC flow against the dep keycloak (uses
|
||||
@ -256,8 +255,8 @@ tests/lasuite-docs/
|
||||
creds to `$CCCI_DEPS_FILE` — BEFORE the recipe deploy.
|
||||
2. Deploy lasuite-docs (`lasu-<6hex>.ci.commoninternet.net`); `install_steps.sh` wires the OIDC
|
||||
env into that one deploy.
|
||||
3. Run install / upgrade / backup / restore + the 3 functional tests against the shared
|
||||
deployment (custom tier).
|
||||
3. Run install / upgrade / backup / restore + the 3 custom tests against the shared
|
||||
deployment (custom tier).
|
||||
4. Teardown lasuite-docs, then the keycloak dep (LAST), both with verify=True.
|
||||
5. Print the run summary; non-zero exit code on any failure (DG4.1 deploy-count mismatch, tier
|
||||
FAIL, dep teardown leak — all surfaced).
|
||||
@ -268,10 +267,10 @@ tests/lasuite-docs/
|
||||
`COMPOSE_FILE=compose.yml:compose.mumbleweb.yml` for the base; `UPGRADE_EXTRA_ENV` adds the
|
||||
native `compose.host-ports.yml` at PR-head so 64738 is host-published on latest; private
|
||||
`_WELCOME_TEXT_MARKER`/`_MAX_USERS` constants; `READY_PROBE(ctx)` TCP 64738 — phase-aware via
|
||||
the live COMPOSE_FILE), `functional/_mumble_proto.py` + the protocol/config-round-trip
|
||||
the live COMPOSE_FILE), `custom/_mumble_proto.py` + the protocol/config-round-trip
|
||||
tests, `ops.py`/`test_backup.py`/`test_restore.py` (sqlite P4). See §2.4.
|
||||
- **Multi-service, dep-less, in-container functional — `tests/mailu/`**: `recipe_meta.py`
|
||||
(`EXTRA_ENV(ctx)` with `TLS_FLAVOR=notls` + `MAIL_DOMAIN`/`HOSTNAMES`/`TRAEFIK_STACK_NAME`),
|
||||
`functional/_mailu.py` (flask-CLI helpers), `test_mailbox.py` (create→config-export read-back),
|
||||
`custom/_mailu.py` (flask-CLI helpers), `test_mailbox.py` (create→config-export read-back),
|
||||
`test_mail_flow.py` (in-container sendmail→doveadm delivery). No backupbot → P4 N/A (PARITY.md +
|
||||
DEFERRED.md). See §2.4.
|
||||
|
||||
@ -22,7 +22,7 @@ A recipe customizes its CI through **three distinct mechanisms**:
|
||||
|---|---|---|
|
||||
| **Declarative settings** | Python assignments in `tests/<recipe>/recipe_meta.py` | `DEPLOY_TIMEOUT = 1500`, `UPGRADE_BASE_VERSION = "2.3.1+..."` |
|
||||
| **Code hooks** | Callables in `recipe_meta.py`, `ops.py` functions, one shell hook | `def READY_PROBE(ctx): ...`, `pre_upgrade(ctx)`, `install_steps.sh` |
|
||||
| **File presence** | A file existing at a discovered path changes behavior | `test_upgrade.py` overlay, `functional/test_*.py`, `compose.ccci.yml` |
|
||||
| **File presence** | A file existing at a discovered path changes behavior | `test_upgrade.py` overlay, `custom/test_*.py`, `compose.ccci.yml` |
|
||||
|
||||
There is additionally a fourth, **operator-facing, local-dev-only** surface: environment variables
|
||||
(`CCCI_SKIP_GENERIC*`) that suppress the generic floor at run time (§7). Whatever a run resolves
|
||||
@ -60,15 +60,15 @@ tests/<recipe>/ # cc-ci side (repo-local mirrors the same s
|
||||
├── recipe_meta.py # THE config file: registry-validated keys + ctx-hooks (§4)
|
||||
├── test_<op>.py # lifecycle overlay assertions, op ∈ install|upgrade|backup|restore (§5.1)
|
||||
├── ops.py # pre_<op>(ctx) seed hooks (§5.2)
|
||||
├── functional/test_*.py # custom tier: parity ports + recipe-specific (§5.3)
|
||||
├── playwright/test_*.py # custom tier: UI flows (§5.3)
|
||||
├── custom/test_*.py # custom tier: parity ports + recipe-specific + UI flows (§5.3)
|
||||
├── install_steps.sh # pre-deploy shell hook (the ONLY shell hook) (§5.4)
|
||||
├── compose.ccci.yml # CI-only compose overlay (first-class) (§5.5)
|
||||
└── PARITY.md # enrollment contract doc (human-read only)
|
||||
```
|
||||
|
||||
**Placement rule (custom tests):** ALL custom-tier tests live under `functional/` or
|
||||
`playwright/`. A top-level `test_*.py` is a lifecycle overlay (`test_<op>.py`) and nothing else —
|
||||
**Placement rule (custom tests):** ALL custom-tier tests live under canonical `custom/`.
|
||||
Deprecated `functional/` and `playwright/` aliases are still discovered with a loud warning so
|
||||
coverage is not silently lost while recipe trees migrate. A top-level `test_*.py` is a lifecycle overlay (`test_<op>.py`) and nothing else —
|
||||
top-level non-lifecycle files are NOT discovered (`discovery.custom_tests`; the lifecycle-name
|
||||
exclusion stays as a safety net so a misfiled `test_<op>.py` can never double-run).
|
||||
|
||||
@ -76,7 +76,8 @@ Precedence (machine-docs/DECISIONS.md, implemented in `discovery.py`):
|
||||
|
||||
- lifecycle overlay `test_<op>.py`: repo-local **wins** over cc-ci (same-name collision); the
|
||||
generic floor still runs additively alongside.
|
||||
- custom tier (`functional/` + `playwright/`): **ALL** run, from both locations (no collision
|
||||
- custom tier (`custom/`, plus deprecated alias dirs during migration): **ALL** run, from both
|
||||
locations (no collision
|
||||
concept).
|
||||
- `install_steps.sh`: repo-local > cc-ci, or none.
|
||||
- `ops.py` pre-op hook: cc-ci wins; repo-local consulted only if approved.
|
||||
@ -181,15 +182,16 @@ def pre_restore(ctx): _psql(ctx.domain, "DROP TABLE ci_marker") # damage, rest
|
||||
Seed → op → assert is the whole pattern: `pre_backup` writes a marker, the orchestrator backs up,
|
||||
`pre_restore` destroys it, the orchestrator restores, `test_restore.py` asserts the marker is back.
|
||||
|
||||
### 5.3 Custom tier — `functional/` and `playwright/` ONLY
|
||||
### 5.3 Custom tier — canonical `custom/`
|
||||
|
||||
All custom-tier tests live under `tests/<recipe>/functional/` or `tests/<recipe>/playwright/`
|
||||
(discovery: `discovery.custom_tests`; the placement rule, §3). Run in the CUSTOM tier, after
|
||||
All custom-tier tests live under `tests/<recipe>/custom/` (discovery: `discovery.custom_tests`;
|
||||
the placement rule, §3). Deprecated `functional/` and `playwright/` dirs are still recognized
|
||||
with a warning during the migration window. Custom tests run in the CUSTOM tier, after
|
||||
restore, against the post-upgrade (PR-head) app. ALL discovered files run — cc-ci's and (if
|
||||
HC2-approved) repo-local's, additively.
|
||||
|
||||
Enrollment contract (`docs/enroll-recipe.md`): ≥2 NEW functional tests beyond ports of existing
|
||||
upstream checks; ported tests carry `SOURCE:` comments. Playwright tests get the shared
|
||||
Enrollment contract (`docs/enroll-recipe.md`): ≥2 NEW custom tests beyond ports of existing
|
||||
upstream checks; ported tests carry `SOURCE:` comments. Browser-driven custom tests get the shared
|
||||
browser/harness helpers (`harness.browser`); SSO recipes get `harness.sso`
|
||||
(`setup_keycloak_realm` — idempotent, `oidc_password_grant` — provider-pluggable). The documented
|
||||
import toolbox for custom tests is `from harness import lifecycle, sso, browser`.
|
||||
@ -268,7 +270,7 @@ deploy BASE (UPGRADE_BASE_VERSION or recipe_versions[-2]; EXTRA_ENV; install_ste
|
||||
→ BACKUP tier
|
||||
→ pre_restore(ctx) → restore
|
||||
→ RESTORE tier
|
||||
→ CUSTOM tier (functional/ + playwright/; deps via the `deps` fixture)
|
||||
→ CUSTOM tier (custom/; deps via the `deps` fixture)
|
||||
→ SCREENSHOT (best-effort, never affects the verdict)
|
||||
→ teardown (deps LAST)
|
||||
```
|
||||
@ -293,7 +295,7 @@ RECIPE=<recipe> PR=<n> REF=<sha> SRC=recipe-maintainers/<recipe> \
|
||||
meta (non-default): DEPLOY_TIMEOUT=1500 DEPS=['keycloak'] EXTRA_ENV='<hook>'
|
||||
hooks: ops.py[pre_backup,pre_upgrade](cc-ci) install_steps.sh(cc-ci) compose.ccci.yml(cc-ci)
|
||||
overlays: test_backup.py(cc-ci) test_restore.py(repo-local)
|
||||
custom tests: functional/=5 playwright/=2 (cc-ci)
|
||||
custom tests: custom/=7 (cc-ci)
|
||||
env overrides: (none)
|
||||
```
|
||||
|
||||
|
||||
@ -114,11 +114,12 @@ repo-local <recipe-repo>/tests/test_<op>.py (upstream-authoritative; gated
|
||||
Only ONE overlay source wins for a given op (repo-local > cc-ci); the generic floor runs **in
|
||||
addition** unless explicitly opted out.
|
||||
|
||||
**Custom (non-lifecycle) tests** — e.g. `functional/test_sso.py` — are **opt-in and additive**:
|
||||
**Custom (non-lifecycle) tests** — e.g. `custom/test_sso.py` — are **opt-in and additive**:
|
||||
they have no generic equivalent and run only when present, discovered from both locations
|
||||
(repo-local gated by the HC2 allowlist). Placement rule: custom tests live ONLY under
|
||||
`functional/` or `playwright/`; a top-level `test_*.py` is a lifecycle overlay and nothing else
|
||||
(top-level non-lifecycle files are not discovered).
|
||||
(repo-local gated by the HC2 allowlist). Placement rule: custom tests live under canonical
|
||||
`custom/`; deprecated `functional/` and `playwright/` aliases are still discovered with a loud
|
||||
warning so old recipe trees are not silently dropped. A top-level `test_*.py` is a lifecycle
|
||||
overlay and nothing else (top-level non-lifecycle files are not discovered).
|
||||
|
||||
### Pre-op seed hooks (per-recipe `ops.py`)
|
||||
|
||||
|
||||
@ -4,11 +4,11 @@
|
||||
(Builder-only section — read-only to Adversary)
|
||||
|
||||
- [x] Seed `STATUS-cfold.md` + `JOURNAL-cfold.md`; consume Adversary inbox
|
||||
- [ ] Record deprecated-folder policy in `DECISIONS.md`
|
||||
- [ ] Update discovery + manifest to make `custom/` canonical without silent coverage loss
|
||||
- [ ] Update unit tests for discovery/manifest behavior and ordering
|
||||
- [ ] Migrate all cc-ci custom tests/helper modules into `tests/<recipe>/custom/`
|
||||
- [ ] Update docs (`docs/recipe-customization.md`, `docs/testing.md`, `docs/enroll-recipe.md`)
|
||||
- [x] Record deprecated-folder policy in `DECISIONS.md`
|
||||
- [x] Update discovery + manifest to make `custom/` canonical without silent coverage loss
|
||||
- [x] Update unit tests for discovery/manifest behavior and ordering
|
||||
- [x] Migrate all cc-ci custom tests/helper modules into `tests/<recipe>/custom/`
|
||||
- [x] Update docs (`docs/recipe-customization.md`, `docs/testing.md`, `docs/enroll-recipe.md`)
|
||||
- [ ] Produce M1 coverage-diff proof: discovered custom-test set identical before/after
|
||||
- [ ] Claim M1 with WHAT/HOW/EXPECTED/WHERE in `STATUS-cfold.md`
|
||||
- [ ] Build the pre-sweep recipe baseline matrix for M2
|
||||
|
||||
@ -4,6 +4,13 @@ Architecture decisions and dead-ends. One line of rationale each. (§0, §8)
|
||||
|
||||
## Settled
|
||||
|
||||
- **cfold deprecated-folder policy — SETTLED (2026-06-12, phase cfold).** `tests/<recipe>/custom/`
|
||||
is the canonical home for custom tests. Discovery keeps recognizing legacy `functional/` and
|
||||
`playwright/` subdirs for both cc-ci and approved repo-local tests as a temporary compatibility
|
||||
alias, but it emits a one-line warning to stderr whenever it discovers tests there. Rationale:
|
||||
the phase plan forbids silent coverage loss, and recipe repos outside this clone may still be on
|
||||
the old layout during the migration window.
|
||||
|
||||
- **Wildcard TLS:** operator pre-issues wildcard cert at `/var/lib/ci-certs/live/`; Traefik file
|
||||
provider serves it; **no ACME** for commoninternet.net. (Plan §4.0/§8 — fixed.)
|
||||
- **Repo:** `git.autonomic.zone/recipe-maintainers/cc-ci`, private. Bot is org admin. (Bootstrap.)
|
||||
|
||||
@ -1,44 +1,59 @@
|
||||
# JOURNAL — phase `cfold` (Builder)
|
||||
# JOURNAL — phase cfold
|
||||
|
||||
Design rationale, investigations, and dead-ends. Adversary does NOT read this before
|
||||
forming its verdict (anti-anchoring per plan §6.1). See STATUS-cfold.md for claim context.
|
||||
## 2026-06-11 — Phase cfold start
|
||||
|
||||
---
|
||||
### Investigation findings
|
||||
|
||||
## 2026-06-12 — bootstrap + initial orient
|
||||
Pre-existing test layout:
|
||||
- 60 files in `functional/` subdirs across 20 recipes
|
||||
- 4 files in `playwright/` subdirs (cryptpad, custom-html, uptime-kuma)
|
||||
- Helper modules to move: `_discourse.py`, `_ghost.py`, `_mailu.py`, `_mm.py`, `_mumble_proto.py`, `drone/functional/__init__.py`
|
||||
- `mailu/test_backup.py`, `test_restore.py`, `ops.py` explicitly add `functional/` to sys.path — need updating to `custom/`
|
||||
|
||||
Read in full:
|
||||
- `/srv/cc-ci/cc-ci-plan/plan-phase-cfold-custom-folder.md`
|
||||
- `/srv/cc-ci/cc-ci-plan/plan.md` bootstrap plus §§6.1, 7, 9
|
||||
### Decision: deprecated aliases
|
||||
|
||||
Initial repo/phase state after `git pull --rebase`:
|
||||
- pulled Adversary updates `574306e -> 87566b1`
|
||||
- `machine-docs/BACKLOG-cfold.md` and `machine-docs/REVIEW-cfold.md` existed already
|
||||
- `machine-docs/STATUS-cfold.md` and `machine-docs/JOURNAL-cfold.md` were missing
|
||||
Per plan §2 option (RECOMMENDED): keep recognizing `functional/`/`playwright/` as deprecated aliases
|
||||
AND emit a loud one-line warning when a test is found in a deprecated folder. Using `warnings.warn()`
|
||||
at import time of discovery or `print()` directly. Will use `print()` (stderr) so it shows up in CI
|
||||
logs without needing to configure warning filters.
|
||||
|
||||
Bootstrap checks run from this clone:
|
||||
Implementation: `subdirs = ("custom", "functional", "playwright")` — canonical first — and after
|
||||
finding a test in `functional/` or `playwright/`, emit:
|
||||
`print(f"WARNING [cfold]: test found in deprecated folder '{sub}/' — move to custom/: {path}", flush=True, file=sys.stderr)`
|
||||
|
||||
This way:
|
||||
- `custom/` is canonical and gets discovered first
|
||||
- Old folders still work (zero breakage for repo-local tests) but emit a loud warning
|
||||
- No silent coverage loss possible
|
||||
|
||||
## 2026-06-12 — M1 checkpoint: canonical `custom/` layout landed locally
|
||||
|
||||
Code/work completed:
|
||||
- `runner/harness/discovery.py`: canonical `custom/` discovery, deprecated alias warnings, and
|
||||
`custom_subdir_label()` normalization helper.
|
||||
- `runner/harness/manifest.py`: custom-test counts now normalize to canonical `custom`.
|
||||
- all cc-ci custom tests/helper modules moved from `tests/<recipe>/{functional,playwright}/` into
|
||||
`tests/<recipe>/custom/`.
|
||||
- helper-import fallout fixed where needed (`tests/mailu/{ops.py,test_backup.py,test_restore.py}`).
|
||||
- docs updated to describe `custom/` as the canonical layout and explain the alias-compatibility window.
|
||||
|
||||
Mechanical move summary:
|
||||
- 64 custom test files relocated into `custom/`
|
||||
- helper modules relocated too: `_discourse.py`, `_ghost.py`, `_mailu.py`, `_mm.py`,
|
||||
`_mumble_proto.py`, `tests/drone/custom/__init__.py`
|
||||
|
||||
Verification:
|
||||
```bash
|
||||
ssh cc-ci 'hostname && whoami && nixos-version'
|
||||
# nixos
|
||||
# root
|
||||
# 24.11.20250630.50ab793 (Vicuna)
|
||||
|
||||
set -a && . /srv/cc-ci/.testenv && set +a && curl -s "https://$GITEA_URL/api/v1/version"
|
||||
# {"version":"1.24.2"}
|
||||
|
||||
getent hosts "probe-$RANDOM.ci.commoninternet.net"
|
||||
# 91.98.47.73 probe-22588.ci.commoninternet.net
|
||||
nix shell nixpkgs#python312Packages.pytest --command pytest \
|
||||
tests/unit/test_discovery.py tests/unit/test_discovery_phase2.py tests/unit/test_manifest.py -q
|
||||
# ..................
|
||||
# 18 passed in 0.09s
|
||||
```
|
||||
|
||||
Initial cfold code scan confirms the planned touch points are still unmigrated:
|
||||
- `runner/harness/discovery.py` still globs `("functional", "playwright")`
|
||||
- `runner/harness/manifest.py` still reports subdir names verbatim
|
||||
- unit tests still build fixtures under `functional/` and `playwright/`
|
||||
- repo grep still finds many folder-name references in docs/tests and the recipe trees themselves
|
||||
Post-move grep state:
|
||||
- remaining `functional/` / `playwright/` matches in live code are intentional: alias-policy docs,
|
||||
deprecated-folder assertions in the unit tests, and discovery comments describing the alias behavior.
|
||||
- the pre-migration inventory in `BACKLOG-cfold.md` is intentionally unchanged because it is the M1
|
||||
baseline record the Adversary will compare against.
|
||||
|
||||
Adversary inbox/review updates at 2026-06-12T00:00Z and 2026-06-12T16:00Z were procedural only:
|
||||
no claim pending, phase status file missing on `origin/main`. Consuming
|
||||
`machine-docs/BUILDER-INBOX.md` in the same commit that seeds cfold state.
|
||||
|
||||
Next: implement the smallest M1 slice first: discovery + alias policy + unit/manifest updates,
|
||||
then migrate the recipe trees and docs, then assemble the before/after coverage proof.
|
||||
Next: assemble the before/after discovery proof so M1 can be claimed without hand-waving.
|
||||
|
||||
@ -1,25 +1,54 @@
|
||||
# STATUS — phase `cfold` (collapse custom-test folders)
|
||||
# STATUS — phase cfold (custom-folder collapse)
|
||||
|
||||
SSOT: `/srv/cc-ci/cc-ci-plan/plan-phase-cfold-custom-folder.md`
|
||||
**Phase:** cfold — collapse `functional/`+`playwright/` into `custom/`
|
||||
**Builder:** autonomic-bot
|
||||
**Updated:** 2026-06-11
|
||||
|
||||
## Current state
|
||||
---
|
||||
|
||||
- Phase bootstrapped on `main` after reading the phase plan and `plan.md` §§1, 6.1, 7, 9.
|
||||
- Access/bootstrap checks passed from this clone:
|
||||
- `ssh cc-ci 'hostname && whoami && nixos-version'` -> `nixos`, `root`, `24.11.20250630.50ab793 (Vicuna)`
|
||||
- `curl -s "https://$GITEA_URL/api/v1/version"` -> `{"version":"1.24.2"}`
|
||||
- `getent hosts "probe-$RANDOM.ci.commoninternet.net"` -> wildcard DNS resolves (`91.98.47.73` in the bootstrap probe)
|
||||
- Adversary notes at `REVIEW-cfold.md` 2026-06-12T00:00Z and 2026-06-12T16:00Z processed: both were procedural only (`STATUS-cfold.md` missing on `origin/main`); this file now exists and will land with the inbox-consumption commit.
|
||||
- Current work item: M1 implementation start
|
||||
- decide and record deprecated-folder behavior (`custom/` canonical, no silent coverage loss)
|
||||
- update discovery/manifest/unit tests/docs for `custom/`
|
||||
- migrate cc-ci custom tests and helper modules from `functional/` + `playwright/` into `custom/`
|
||||
- produce coverage-diff proof for the Adversary before any M1 claim
|
||||
## M1 — IN PROGRESS
|
||||
|
||||
## Gate
|
||||
Completed in this checkpoint:
|
||||
- discovery.py: `custom/` canonical + deprecated aliases with warnings
|
||||
- `git mv` all 64 custom tests (60 functional + 4 playwright) across 20 recipes
|
||||
- helper modules moved alongside their tests into `custom/`
|
||||
- sys.path refs updated in mailu lifecycle overlays
|
||||
- docs updated (`README.md`, `recipe-customization.md`, `testing.md`, `enroll-recipe.md`)
|
||||
- unit tests updated (`test_discovery.py`, `test_discovery_phase2.py`, `test_manifest.py`)
|
||||
- manifest.py now reports canonical `custom` counts
|
||||
|
||||
No gate claimed yet.
|
||||
Verification so far:
|
||||
- `nix shell nixpkgs#python312Packages.pytest --command pytest tests/unit/test_discovery.py tests/unit/test_discovery_phase2.py tests/unit/test_manifest.py -q`
|
||||
- Expected/current: `18 passed`
|
||||
|
||||
## Blocked
|
||||
Remaining before an M1 claim:
|
||||
- assemble a cold-verifiable before/after coverage proof (same discovered custom-test set, paths renamed only)
|
||||
- write WHAT/HOW/EXPECTED/WHERE into this file for the Adversary
|
||||
|
||||
(nothing)
|
||||
---
|
||||
|
||||
## Baseline (pre-cfold) — custom test count per recipe
|
||||
|
||||
| Recipe | Count |
|
||||
|--------|-------|
|
||||
| bluesky-pds | 4 |
|
||||
| cryptpad | 4 |
|
||||
| custom-html | 4 |
|
||||
| custom-html-tiny | 1 |
|
||||
| discourse | 3 |
|
||||
| drone | 1 |
|
||||
| ghost | 4 |
|
||||
| hedgedoc | 2 |
|
||||
| immich | 3 |
|
||||
| keycloak | 3 |
|
||||
| lasuite-docs | 5 |
|
||||
| lasuite-drive | 3 |
|
||||
| lasuite-meet | 3 |
|
||||
| mailu | 3 |
|
||||
| matrix-synapse | 3 |
|
||||
| mattermost-lts | 3 |
|
||||
| mumble | 5 |
|
||||
| n8n | 4 |
|
||||
| plausible | 2 |
|
||||
| uptime-kuma | 4 |
|
||||
| **TOTAL** | **64** |
|
||||
|
||||
@ -11,8 +11,8 @@ hook; the orchestrator decides additive-vs-skip. Sources, in precedence order
|
||||
> cc-ci tests/<recipe>/test_<op>.py
|
||||
(the generic tests/_generic/test_<op>.py is the always-present floor, run separately by default)
|
||||
|
||||
custom test_*.py (functional/ + playwright/ ONLY, rcust P4 placement rule) — ALL run,
|
||||
additively, from BOTH locations (opt-in).
|
||||
custom test_*.py (`custom/` canonical; `functional/` + `playwright/` deprecated aliases) —
|
||||
ALL run, additively, from BOTH locations (opt-in).
|
||||
|
||||
install-steps hook — install_steps.sh: repo-local > cc-ci, or none.
|
||||
|
||||
@ -27,6 +27,7 @@ from __future__ import annotations
|
||||
|
||||
import glob
|
||||
import os
|
||||
import sys
|
||||
|
||||
LIFECYCLE_OPS = ("install", "upgrade", "backup", "restore")
|
||||
|
||||
@ -102,15 +103,16 @@ def resolve_op(recipe: str, op: str, repo_local_dir: str | None) -> tuple[str, s
|
||||
|
||||
def custom_tests(recipe: str, repo_local_dir: str | None) -> list[tuple[str, str]]:
|
||||
"""All custom-tier test_*.py from cc-ci's tests/<recipe>/ and (if approved) the recipe's
|
||||
repo-local tests/. PLACEMENT RULE (rcust P4): custom tests live ONLY under
|
||||
- functional/ tests/<recipe>/functional/test_*.py (parity ports + recipe-specific)
|
||||
- playwright/ tests/<recipe>/playwright/test_*.py (UI flows)
|
||||
repo-local tests/. PLACEMENT RULE (rcust P4): custom tests live under canonical
|
||||
- custom/ tests/<recipe>/custom/test_*.py (canonical home)
|
||||
- functional/ tests/<recipe>/functional/test_*.py (deprecated alias)
|
||||
- playwright/ tests/<recipe>/playwright/test_*.py (deprecated alias)
|
||||
A top-level test_*.py is a LIFECYCLE OVERLAY (test_<op>.py) and nothing else — top-level
|
||||
non-lifecycle files are NOT discovered (zero users at the time of the change; the lifecycle-
|
||||
name exclusion below stays as a safety net so a misfiled test_<op>.py can never double-run).
|
||||
Repo-local is consulted only for allowlist-approved recipes (HC2)."""
|
||||
lifecycle_names = {f"test_{op}.py" for op in LIFECYCLE_OPS}
|
||||
subdirs = ("functional", "playwright")
|
||||
subdirs = ("custom", "functional", "playwright")
|
||||
found: list[tuple[str, str]] = []
|
||||
for source, d in (("cc-ci", cc_ci_dir(recipe)), ("repo-local", _gated(recipe, repo_local_dir))):
|
||||
if not d or not os.path.isdir(d):
|
||||
@ -118,6 +120,12 @@ def custom_tests(recipe: str, repo_local_dir: str | None) -> list[tuple[str, str
|
||||
for sub in subdirs:
|
||||
for p in sorted(glob.glob(os.path.join(d, sub, "test_*.py"))):
|
||||
if os.path.basename(p) not in lifecycle_names:
|
||||
if sub != "custom":
|
||||
print(
|
||||
f"WARNING [cfold]: test found in deprecated folder '{sub}/' — move to custom/: {p}",
|
||||
file=sys.stderr,
|
||||
flush=True,
|
||||
)
|
||||
found.append((source, p))
|
||||
return found
|
||||
|
||||
|
||||
@ -52,7 +52,7 @@ def _pre_ops(path: str) -> list[str]:
|
||||
def _custom_counts(recipe: str, repo_local: str | None) -> dict[str, dict[str, int]]:
|
||||
out: dict[str, dict[str, int]] = {}
|
||||
for source, path in discovery.custom_tests(recipe, repo_local):
|
||||
sub = os.path.basename(os.path.dirname(path)) # functional | playwright
|
||||
sub = "custom"
|
||||
out.setdefault(source, {}).setdefault(sub, 0)
|
||||
out[source][sub] += 1
|
||||
return out
|
||||
|
||||
@ -4,8 +4,8 @@ Phase-2 P2 mapping table.
|
||||
|
||||
| recipe-maintainer file | cc-ci file | what's verified | status |
|
||||
|---|---|---|---|
|
||||
| (no health_check.py in the recipe-maintainer corpus) | `tests/bluesky-pds/functional/test_health_check.py` | GETs `/xrpc/_health` (the PDS health endpoint); asserts 200 + JSON with `version` field. Phase-2 health_check aligned with the parity-port convention. | **Phase-2 health_check** |
|
||||
| `recipe-info/bluesky-pds/tests/goat_account.py` | `tests/bluesky-pds/functional/test_account_and_post.py` | Original: `goat pds describe`, list/cleanup, account create, verify listed, delete, verify gone. cc-ci port preserves the account-lifecycle assertions + adds an **atproto post round-trip** (createSession→createRecord→getRecord, asserts post text round-trips) — the §4.3 prescribed test ("create a test account (goat CLI), create a post via atproto, fetch it back, delete the account"). F2-8 closed. | **ported** |
|
||||
| (no health_check.py in the recipe-maintainer corpus) | `tests/bluesky-pds/custom/test_health_check.py` | GETs `/xrpc/_health` (the PDS health endpoint); asserts 200 + JSON with `version` field. Phase-2 health_check aligned with the parity-port convention. | **Phase-2 health_check** |
|
||||
| `recipe-info/bluesky-pds/tests/goat_account.py` | `tests/bluesky-pds/custom/test_account_and_post.py` | Original: `goat pds describe`, list/cleanup, account create, verify listed, delete, verify gone. cc-ci port preserves the account-lifecycle assertions + adds an **atproto post round-trip** (createSession→createRecord→getRecord, asserts post text round-trips) — the §4.3 prescribed test ("create a test account (goat CLI), create a post via atproto, fetch it back, delete the account"). F2-8 closed. | **ported** |
|
||||
|
||||
## Recipe-specific tests (Phase-2 P3, ≥2 beyond parity)
|
||||
|
||||
@ -14,8 +14,8 @@ public XRPC API + the well-known `atproto-did` server identifier. Two new functi
|
||||
|
||||
| cc-ci file | what's verified | rationale |
|
||||
|---|---|---|
|
||||
| `tests/bluesky-pds/functional/test_describe_server.py` | GETs `/xrpc/com.atproto.server.describeServer` (the public atproto endpoint that advertises the PDS's available account creation policy); asserts 200 + JSON envelope with at least `availableUserDomains` (array; the PDS's hosting domains) or `inviteCodeRequired` (bool). | Proves the atproto XRPC API is alive AND the PDS-specific configuration is being served (not just a generic 200). Non-vacuous: a PDS that boots but can't serve its server description is broken. |
|
||||
| `tests/bluesky-pds/functional/test_session_auth.py` | GETs `/xrpc/com.atproto.server.getSession` (no auth); asserts **401** + a JSON XRPC error envelope with an `error` field. | Proves the PDS's atproto auth contract is enforced. Non-vacuous: 200 = anonymous leak (security bug); 404 = route missing; 5xx = backend broken — only 401 + a proper XRPC error envelope indicates a correctly-wired PDS. (An earlier draft tried `/.well-known/atproto-did` but that endpoint is only published when the bare DOMAIN is registered as a server-DID, which the recipe doesn't auto-configure.) |
|
||||
| `tests/bluesky-pds/custom/test_describe_server.py` | GETs `/xrpc/com.atproto.server.describeServer` (the public atproto endpoint that advertises the PDS's available account creation policy); asserts 200 + JSON envelope with at least `availableUserDomains` (array; the PDS's hosting domains) or `inviteCodeRequired` (bool). | Proves the atproto XRPC API is alive AND the PDS-specific configuration is being served (not just a generic 200). Non-vacuous: a PDS that boots but can't serve its server description is broken. |
|
||||
| `tests/bluesky-pds/custom/test_session_auth.py` | GETs `/xrpc/com.atproto.server.getSession` (no auth); asserts **401** + a JSON XRPC error envelope with an `error` field. | Proves the PDS's atproto auth contract is enforced. Non-vacuous: 200 = anonymous leak (security bug); 404 = route missing; 5xx = backend broken — only 401 + a proper XRPC error envelope indicates a correctly-wired PDS. (An earlier draft tried `/.well-known/atproto-did` but that endpoint is only published when the bare DOMAIN is registered as a server-DID, which the recipe doesn't auto-configure.) |
|
||||
|
||||
Two specific tests + parity health_check = ≥2 floor met. Backup data-integrity is N/A unless the
|
||||
recipe declares `backupbot.backup=true` labels (Phase-1d auto-detect handles the skip).
|
||||
|
||||
@ -5,7 +5,7 @@ Phase-2 P2 mapping table. The Adversary cold-verifies parity by reading the sour
|
||||
|
||||
| recipe-maintainer file | cc-ci file | what's verified | status |
|
||||
|---|---|---|---|
|
||||
| `recipe-info/cryptpad/tests/health_check.py` | `tests/cryptpad/functional/test_health_check.py` | HTTP 200 from the served root. The cc-ci port preserves the assertion shape adapted to the ephemeral per-run domain. | **ported** |
|
||||
| `recipe-info/cryptpad/tests/health_check.py` | `tests/cryptpad/custom/test_health_check.py` | HTTP 200 from the served root. The cc-ci port preserves the assertion shape adapted to the ephemeral per-run domain. | **ported** |
|
||||
| `recipe-info/cryptpad/tests/oidc_login.py` | (Q3.4 follow-up — needs cryptpad OIDC env wired to the dep authentik) | The original is a cross-recipe authenticated flow against **authentik** (not keycloak). The cc-ci port requires: (1) Q2.2 authentik enrollment + `setup_authentik_realm` harness backend, (2) cryptpad's install_steps.sh wiring the dep authentik's client_secret + OIDC env. Both are tracked Q5 catch-up items. | **deferred** |
|
||||
|
||||
## Recipe-specific tests (Phase-2 P3, ≥2 beyond parity)
|
||||
@ -17,9 +17,9 @@ object + read-it-back" test (plan §4.3 floor) MUST use a real browser (per plan
|
||||
|
||||
| cc-ci file | what's verified | rationale |
|
||||
|---|---|---|
|
||||
| `tests/cryptpad/playwright/test_pad_content_roundtrip.py` | **§4.3 create-an-object + read-it-back (resolves F2-9).** Opens `/pad/` → CryptPad auto-creates a fragment-keyed pad (`#/2/pad/edit/<key>/`); types a unique marker into the CKEditor rich-text body (nested sandbox iframe `…/pad/ckeditor-inner.html`); waits for the encrypted update to sync ("Saved"); then opens a **brand-new browser context** (no shared localStorage/cookies) and navigates to the captured pad URL; asserts the marker is present in the re-decrypted body. | Phase 2 P3/§4.3 floor — proves genuine **end-to-end-encrypted persistence**: the fresh session carries only the URL (incl. its fragment key), so a successful read-back means the content was persisted server-side as ciphertext and correctly decrypted by a new client. Not a health/SPA stand-in. Mapped empirically against CryptPad 2026.2.0 (editor in a deep nested frame; ~15s cold-cache LESS-compile init; transient `net::ERR_NETWORK_CHANGED` handled by the shared `goto_with_retry` + a mid-load reload retry). |
|
||||
| `tests/cryptpad/playwright/test_pad_create.py` | Browses to `/`. Asserts SPA branding present in the rendered title/body, canonical CryptPad asset paths (`/customize/`, `/components/`, `main.js`, `/api/broadcast`) referenced in the DOM, and no JavaScript console errors during initial load (with `401`/`403`/`favicon` warnings filtered as non-blocking). | Phase 2 P6 — proves CryptPad's SPA renders in a real browser with its JS bundle wired and no fatal client-side errors. (Complements the roundtrip test above; was the "maximal subset" while create-and-read-back was deferred — now superseded by the full roundtrip, kept as a fast SPA-liveness check.) |
|
||||
| `tests/cryptpad/functional/test_spa_assets.py` | GETs `/`; asserts the HTML body contains the **"CryptPad"** brand string AND at least one of CryptPad's canonical asset path references (`/customize/`, `/components/`, `/api/broadcast`, `main.js`). | Distinguishes "the CryptPad SPA bundle is bound and being served" from "nginx is serving an empty default page" (which the parity test alone covers — `/` could 200 from a placeholder). Non-vacuous: a wedged cryptpad-server replaced by a fallback page would 200 but contain none of these markers. |
|
||||
| `tests/cryptpad/custom/test_pad_content_roundtrip.py` | **§4.3 create-an-object + read-it-back (resolves F2-9).** Opens `/pad/` → CryptPad auto-creates a fragment-keyed pad (`#/2/pad/edit/<key>/`); types a unique marker into the CKEditor rich-text body (nested sandbox iframe `…/pad/ckeditor-inner.html`); waits for the encrypted update to sync ("Saved"); then opens a **brand-new browser context** (no shared localStorage/cookies) and navigates to the captured pad URL; asserts the marker is present in the re-decrypted body. | Phase 2 P3/§4.3 floor — proves genuine **end-to-end-encrypted persistence**: the fresh session carries only the URL (incl. its fragment key), so a successful read-back means the content was persisted server-side as ciphertext and correctly decrypted by a new client. Not a health/SPA stand-in. Mapped empirically against CryptPad 2026.2.0 (editor in a deep nested frame; ~15s cold-cache LESS-compile init; transient `net::ERR_NETWORK_CHANGED` handled by the shared `goto_with_retry` + a mid-load reload retry). |
|
||||
| `tests/cryptpad/custom/test_pad_create.py` | Browses to `/`. Asserts SPA branding present in the rendered title/body, canonical CryptPad asset paths (`/customize/`, `/components/`, `main.js`, `/api/broadcast`) referenced in the DOM, and no JavaScript console errors during initial load (with `401`/`403`/`favicon` warnings filtered as non-blocking). | Phase 2 P6 — proves CryptPad's SPA renders in a real browser with its JS bundle wired and no fatal client-side errors. (Complements the roundtrip test above; was the "maximal subset" while create-and-read-back was deferred — now superseded by the full roundtrip, kept as a fast SPA-liveness check.) |
|
||||
| `tests/cryptpad/custom/test_spa_assets.py` | GETs `/`; asserts the HTML body contains the **"CryptPad"** brand string AND at least one of CryptPad's canonical asset path references (`/customize/`, `/components/`, `/api/broadcast`, `main.js`). | Distinguishes "the CryptPad SPA bundle is bound and being served" from "nginx is serving an empty default page" (which the parity test alone covers — `/` could 200 from a placeholder). Non-vacuous: a wedged cryptpad-server replaced by a fallback page would 200 but contain none of these markers. |
|
||||
|
||||
Two specific tests — the ≥2 floor is met. Backup data-integrity is exercised by the Phase-1d/1e
|
||||
lifecycle overlays (`test_backup.py`/`test_restore.py` + `ops.py` — see those files for the
|
||||
@ -27,7 +27,7 @@ marker mechanism + the restore-asserts-pre-mutation pattern).
|
||||
|
||||
## Playwright (P6)
|
||||
|
||||
`tests/cryptpad/playwright/test_pad_create.py` (above) is the canonical browser flow — covers P6
|
||||
`tests/cryptpad/custom/test_pad_create.py` (above) is the canonical browser flow — covers P6
|
||||
in full.
|
||||
|
||||
## Non-ports
|
||||
|
||||
@ -1,13 +1,13 @@
|
||||
# Parity — custom-html
|
||||
|
||||
Phase-2 P2 mapping table: every `references/recipe-maintainer/recipe-info/custom-html/tests/*.py` has
|
||||
a comparable cc-ci test under `tests/custom-html/functional/`, asserting the **same thing** (not just
|
||||
a comparable cc-ci test under `tests/custom-html/custom/`, asserting the **same thing** (not just
|
||||
a renamed file). The Adversary cold-verifies parity by reading the source `recipe-info/<file>` and the
|
||||
cc-ci file side-by-side.
|
||||
|
||||
| recipe-maintainer file | cc-ci file | what's verified | status |
|
||||
|---|---|---|---|
|
||||
| `recipe-info/custom-html/tests/health_check.py` | `tests/custom-html/functional/test_health_check.py` | The app is reachable over HTTPS and returns a successful response (the original asserted HTTP 200 against a persistent instance). The cc-ci port preserves the assertion shape — non-5xx status — and adapts to the ephemeral per-run domain via the `live_app` fixture. | **ported** |
|
||||
| `recipe-info/custom-html/tests/health_check.py` | `tests/custom-html/custom/test_health_check.py` | The app is reachable over HTTPS and returns a successful response (the original asserted HTTP 200 against a persistent instance). The cc-ci port preserves the assertion shape — non-5xx status — and adapts to the ephemeral per-run domain via the `live_app` fixture. | **ported** |
|
||||
|
||||
## Recipe-specific tests (Phase-2 P3, ≥2 beyond parity)
|
||||
|
||||
@ -17,8 +17,8 @@ content, fetch it back"). Two new functional tests beyond parity:
|
||||
|
||||
| cc-ci file | what's verified | rationale |
|
||||
|---|---|---|
|
||||
| `tests/custom-html/functional/test_content_roundtrip.py` | Writes a uniquely-marked content file to the served volume via `lifecycle.exec_in_app` and asserts an HTTPS GET to the corresponding path returns that exact byte content — proves the app serves files written into its served volume, not a static synthetic page. | The recipe IS a content-server: a roundtrip is the canonical proof it works for what it's for. |
|
||||
| `tests/custom-html/functional/test_content_type_header.py` | Writes both an `.html` and a `.txt` marker to the served volume, fetches each, and asserts `Content-Type` reflects the file type (`text/html`, `text/plain`) — proves nginx is properly serving with MIME-typed responses, not just returning bytes. | Distinctive nginx-served behavior — distinguishes a working nginx from a misconfigured one that emits everything as `application/octet-stream`. |
|
||||
| `tests/custom-html/custom/test_content_roundtrip.py` | Writes a uniquely-marked content file to the served volume via `lifecycle.exec_in_app` and asserts an HTTPS GET to the corresponding path returns that exact byte content — proves the app serves files written into its served volume, not a static synthetic page. | The recipe IS a content-server: a roundtrip is the canonical proof it works for what it's for. |
|
||||
| `tests/custom-html/custom/test_content_type_header.py` | Writes both an `.html` and a `.txt` marker to the served volume, fetches each, and asserts `Content-Type` reflects the file type (`text/html`, `text/plain`) — proves nginx is properly serving with MIME-typed responses, not just returning bytes. | Distinctive nginx-served behavior — distinguishes a working nginx from a misconfigured one that emits everything as `application/octet-stream`. |
|
||||
|
||||
Both tests run in the **custom** stage against the same `live_app` shared deployment as the
|
||||
lifecycle overlays — no extra deploy, no extra teardown.
|
||||
@ -32,7 +32,7 @@ via `lifecycle.exec_in_app` (volume-direct, immune to the post-backup serving ra
|
||||
|
||||
## Playwright (P6)
|
||||
|
||||
`tests/custom-html/playwright/test_browser_smoke.py` covers the browser-rendered nginx HTML — already
|
||||
`tests/custom-html/custom/test_browser_smoke.py` covers the browser-rendered nginx HTML — already
|
||||
exercised inline by `tests/custom-html/test_install.py::test_serving_and_content` (lifecycle install
|
||||
overlay), which uses Playwright Chromium to confirm the page renders. The Phase-2 split file is the
|
||||
canonical home for browser-flow coverage and is invoked by the **custom** stage.
|
||||
|
||||
@ -1,7 +1,7 @@
|
||||
"""custom-html — Playwright UI flow (Phase 2 P6).
|
||||
|
||||
The recipe-maintainer corpus did not ship a Playwright test for custom-html — but plan §4.1 names
|
||||
`playwright/` as the canonical home for browser flows where a recipe's core UX is a UI. custom-html
|
||||
The recipe-maintainer corpus did not ship a Playwright test for custom-html — but the cfold layout
|
||||
uses `custom/` as the canonical home for browser flows where a recipe's core UX is a UI. custom-html
|
||||
serves HTML; a browser-rendered fetch (vs raw HTTP) proves the page actually renders and any client-
|
||||
side resources resolve. Distinct from `tests/custom-html/test_install.py` which runs Playwright as
|
||||
part of the lifecycle INSTALL overlay; this file is the standalone Phase-2 custom-stage version, so a
|
||||
@ -19,9 +19,9 @@ Defining behaviors exercised against the live per-run deploy:
|
||||
|
||||
| cc-ci file | what's verified | rationale |
|
||||
|---|---|---|
|
||||
| `functional/test_create_topic.py::test_create_topic_roundtrip` | Bootstraps an admin + API key via Rails in the `app` container (`_discourse.mint_admin`), POSTs `/posts.json` to create a NEW topic with a unique marker in title + body, then GETs `/t/<topic_id>.json` and asserts the title (Discourse `title_prettify`-aware) **and** the unique body marker round-tripped in the first post's `cooked`. | §4.3 "create the app's primary object — a topic — and read it back". Non-vacuous: the marker is unique per run, so a stale/echoed response can't pass; a wedged DB/Rails/posting path fails here even though `/srv/status` returns 200. |
|
||||
| `functional/test_site_basic.py::test_site_json_has_discourse_config` | GETs `/site.json` and asserts a Discourse-specific config structure (e.g. a `categories` list), not a bare 200. | Proves Rails is serving its real site config JSON (a distinctive Discourse structure), distinguishing "the forum backend is up + emitting its API" from "a static/error page at /". |
|
||||
| `functional/test_health_check.py::test_discourse_srv_status_ok` | GETs `/srv/status` and asserts the Discourse readiness signal (Rails serving). | Baseline readiness (parity-aligned health check). |
|
||||
| `custom/test_create_topic.py::test_create_topic_roundtrip` | Bootstraps an admin + API key via Rails in the `app` container (`_discourse.mint_admin`), POSTs `/posts.json` to create a NEW topic with a unique marker in title + body, then GETs `/t/<topic_id>.json` and asserts the title (Discourse `title_prettify`-aware) **and** the unique body marker round-tripped in the first post's `cooked`. | §4.3 "create the app's primary object — a topic — and read it back". Non-vacuous: the marker is unique per run, so a stale/echoed response can't pass; a wedged DB/Rails/posting path fails here even though `/srv/status` returns 200. |
|
||||
| `custom/test_site_basic.py::test_site_json_has_discourse_config` | GETs `/site.json` and asserts a Discourse-specific config structure (e.g. a `categories` list), not a bare 200. | Proves Rails is serving its real site config JSON (a distinctive Discourse structure), distinguishing "the forum backend is up + emitting its API" from "a static/error page at /". |
|
||||
| `custom/test_health_check.py::test_discourse_srv_status_ok` | GETs `/srv/status` and asserts the Discourse readiness signal (Rails serving). | Baseline readiness (parity-aligned health check). |
|
||||
|
||||
Two recipe-specific functional tests (create-topic round-trip + site.json config) + the health check
|
||||
= the ≥2 floor met, with a real create-an-object + read-it-back as the characteristic-behavior test.
|
||||
|
||||
@ -10,7 +10,7 @@
|
||||
# 4. Sets DRONE_USER_CREATE so the gitea ci_admin becomes drone's first admin on login.
|
||||
#
|
||||
# If the deps file is absent or has no gitea entry, drone is still deployed (without SCM wiring);
|
||||
# the functional/test_scm_configured.py test then FAILS, which is the correct signal.
|
||||
# the custom/test_scm_configured.py test then FAILS, which is the correct signal.
|
||||
#
|
||||
# Env supplied by the harness:
|
||||
# CCCI_APP_DOMAIN — the per-run drone app domain
|
||||
|
||||
@ -11,15 +11,15 @@ and a JSON Content/Admin API at `/ghost/api/*`. Defining behaviors exercised:
|
||||
|
||||
| cc-ci file | what's verified | rationale |
|
||||
|---|---|---|
|
||||
| `tests/ghost/functional/test_content_api.py` | GETs `/ghost/api/content/settings/`; asserts 200 with `{"settings": {...}}` envelope OR 401/403 with a Ghost error envelope. | Distinguishes "the ghost-server JS process is up + emitting its API" from "a static themed page is served at /." A wedged Ghost backend → 5xx; misrouted nginx → 404. |
|
||||
| `tests/ghost/functional/test_admin_redirect.py` | GETs `/ghost/`; asserts 200 or 302 + Ghost branding/SPA references in the response (or a redirect to /ghost/#/setup on fresh deploy). | Proves the admin route is wired through the nginx proxy. Distinguishes "admin SPA bound" from "404 (route missing)" or "5xx (broken)." |
|
||||
| `tests/ghost/custom/test_content_api.py` | GETs `/ghost/api/content/settings/`; asserts 200 with `{"settings": {...}}` envelope OR 401/403 with a Ghost error envelope. | Distinguishes "the ghost-server JS process is up + emitting its API" from "a static themed page is served at /." A wedged Ghost backend → 5xx; misrouted nginx → 404. |
|
||||
| `tests/ghost/custom/test_admin_redirect.py` | GETs `/ghost/`; asserts 200 or 302 + Ghost branding/SPA references in the response (or a redirect to /ghost/#/setup on fresh deploy). | Proves the admin route is wired through the nginx proxy. Distinguishes "admin SPA bound" from "404 (route missing)" or "5xx (broken)." |
|
||||
|
||||
Two specific tests + parity health_check = ≥2 floor met.
|
||||
|
||||
## Plan §4.3 prescribed deeper test — AUTHORED (closes DEFERRED ghost create-post)
|
||||
|
||||
§4.3 named "create-a-post round-trip" for ghost. Implemented in
|
||||
`tests/ghost/functional/test_post_roundtrip.py` (helper `functional/_ghost.py`):
|
||||
`tests/ghost/custom/test_post_roundtrip.py` (helper `custom/_ghost.py`):
|
||||
1. Wait for the Admin API healthcheck (`GET /ghost/api/admin/site/` → 200).
|
||||
2. Setup the Ghost owner (POST `/ghost/api/admin/authentication/setup/`, fresh deploy) + establish
|
||||
an admin **session cookie** (POST `/ghost/api/admin/session/`) — cookie-aware stdlib opener,
|
||||
|
||||
@ -2,7 +2,7 @@
|
||||
# Ghost serves an HTML site at `/`; admin UI at `/ghost/`. The first GET to /ghost/ redirects
|
||||
# to the setup wizard (302). Ghost exposes a JSON Content API at /ghost/api/content/ which
|
||||
# requires an API key; the Admin API at /ghost/api/admin/ requires a session/token (see
|
||||
# functional/_ghost.py — version-negotiated, no /v3/ path).
|
||||
# custom/_ghost.py — version-negotiated, no /v3/ path).
|
||||
# State lives in a **MySQL** `ghost` DB (compose `db` service, mysql:8.0) + the `ghost_content`
|
||||
# volume (themes/images) — NOT sqlite. The `db` service is backupbot-labelled with a logical
|
||||
# mysqldump pre-hook; P4 (ops.py + test_{backup,restore,upgrade}.py) seeds a `ci_marker` row there.
|
||||
|
||||
@ -14,8 +14,8 @@ HedgeDoc's defining behaviors:
|
||||
|
||||
| cc-ci file | what's verified | rationale |
|
||||
|---|---|---|
|
||||
| `tests/hedgedoc/functional/test_health_check.py` | `GET /` → 200 or 302 | Proves the app is up and routing through Traefik. A wedged HedgeDoc returns 5xx or no response. |
|
||||
| `tests/hedgedoc/functional/test_branding.py` | `GET /` HTML contains hedgedoc/codimd/hackmd markers OR bundle asset refs | Distinguishes "HedgeDoc is serving its own content" from "fallback page." A misrouted or empty backend lacks these markers. |
|
||||
| `tests/hedgedoc/custom/test_health_check.py` | `GET /` → 200 or 302 | Proves the app is up and routing through Traefik. A wedged HedgeDoc returns 5xx or no response. |
|
||||
| `tests/hedgedoc/custom/test_branding.py` | `GET /` HTML contains hedgedoc/codimd/hackmd markers OR bundle asset refs | Distinguishes "HedgeDoc is serving its own content" from "fallback page." A misrouted or empty backend lacks these markers. |
|
||||
|
||||
## Backup data-integrity
|
||||
|
||||
|
||||
@ -5,7 +5,7 @@ Reference corpus: `references/recipe-maintainer/recipe-info/immich/tests/` (heal
|
||||
## Parity ports
|
||||
| recipe-maintainer test | cc-ci test | what's verified |
|
||||
|---|---|---|
|
||||
| `health_check.py` | `tests/immich/functional/test_health_check.py::test_immich_returns_200` | HTTP 200/301/302 from `/` (immich web SPA served). |
|
||||
| `health_check.py` | `tests/immich/custom/test_health_check.py::test_immich_returns_200` | HTTP 200/301/302 from `/` (immich web SPA served). |
|
||||
|
||||
## Recipe-specific functional tests (P3, ≥2 separate tests — characteristic behavior)
|
||||
1. **`test_asset_upload.py::test_immich_upload_asset_readback_and_thumbnail`** — the §4.3
|
||||
|
||||
@ -5,8 +5,8 @@ Phase-2 P2 mapping table. The Adversary cold-verifies parity by reading the sour
|
||||
|
||||
| recipe-maintainer file | cc-ci file | what's verified | status |
|
||||
|---|---|---|---|
|
||||
| `recipe-info/keycloak/tests/health_check.py` | `tests/keycloak/functional/test_health_check.py` | The keycloak master realm endpoint (`/realms/master`) returns HTTP 200 — the original's assertion shape, preserved. The cc-ci port adapts to the ephemeral per-run domain via the `live_app` fixture. | **ported** |
|
||||
| `recipe-info/keycloak/tests/oidc_integration.py` | (deferred to Q3 lasuite-docs) | The original is a **cross-recipe** integration test: it expects both keycloak AND lasuite-docs deployed, with a pre-seeded credentials TOML, and proves a keycloak-issued token is accepted by lasuite-docs. This requires the Phase-2 dependency resolver (Q0.4/Q2.3) + lasuite-docs Phase-2 enrollment. Will land as `tests/lasuite-docs/functional/test_oidc_with_keycloak.py` in Q3, sharing the SSO-setup harness. | **deferred to Q3** (logged in DECISIONS.md) |
|
||||
| `recipe-info/keycloak/tests/health_check.py` | `tests/keycloak/custom/test_health_check.py` | The keycloak master realm endpoint (`/realms/master`) returns HTTP 200 — the original's assertion shape, preserved. The cc-ci port adapts to the ephemeral per-run domain via the `live_app` fixture. | **ported** |
|
||||
| `recipe-info/keycloak/tests/oidc_integration.py` | (deferred to Q3 lasuite-docs) | The original is a **cross-recipe** integration test: it expects both keycloak AND lasuite-docs deployed, with a pre-seeded credentials TOML, and proves a keycloak-issued token is accepted by lasuite-docs. This requires the Phase-2 dependency resolver (Q0.4/Q2.3) + lasuite-docs Phase-2 enrollment. Will land as `tests/lasuite-docs/custom/test_oidc_with_keycloak.py` in Q3, sharing the SSO-setup harness. | **deferred to Q3** (logged in DECISIONS.md) |
|
||||
|
||||
## Recipe-specific tests (Phase-2 P3, ≥2 beyond parity)
|
||||
|
||||
@ -16,8 +16,8 @@ marker realm across upgrade/backup/restore — Phase 1d/1e). Two new functional
|
||||
|
||||
| cc-ci file | what's verified | rationale |
|
||||
|---|---|---|
|
||||
| `tests/keycloak/functional/test_password_grant_token.py` | Obtains an admin-CLI access token via the password grant (`grant_type=password`) against `/realms/master/protocol/openid-connect/token`, asserts the token is a valid JWT (3 base64url-encoded segments), decodes the payload, and asserts the JWT claims include `iss` matching the live domain, `azp == "admin-cli"`, `typ == "Bearer"`, and a future `exp`. | The defining keycloak behavior is issuing JWTs; this test does the canonical password-grant flow against the real running keycloak (real admin user, real password from the abra-generated secret) and proves the JWT contract is intact. Non-vacuous: a wrongly-configured realm, broken signing key, or wrong issuer would fail the claim assertions. |
|
||||
| `tests/keycloak/functional/test_create_client_and_use.py` | Authenticates as admin → creates a confidential client in the master realm via admin API with a known `clientId` + a known client secret → obtains a token via `grant_type=client_credentials` for that client → asserts the token's `azp` (authorized party) matches the new client's clientId → deletes the client (idempotent cleanup). | Proves the full lifecycle of admin-API client creation + service-account token issuance, the canonical "real app integrating with keycloak" flow. Non-vacuous: tests TWO grant types (password + client-credentials) and the admin-API CRUD on clients. |
|
||||
| `tests/keycloak/custom/test_password_grant_token.py` | Obtains an admin-CLI access token via the password grant (`grant_type=password`) against `/realms/master/protocol/openid-connect/token`, asserts the token is a valid JWT (3 base64url-encoded segments), decodes the payload, and asserts the JWT claims include `iss` matching the live domain, `azp == "admin-cli"`, `typ == "Bearer"`, and a future `exp`. | The defining keycloak behavior is issuing JWTs; this test does the canonical password-grant flow against the real running keycloak (real admin user, real password from the abra-generated secret) and proves the JWT contract is intact. Non-vacuous: a wrongly-configured realm, broken signing key, or wrong issuer would fail the claim assertions. |
|
||||
| `tests/keycloak/custom/test_create_client_and_use.py` | Authenticates as admin → creates a confidential client in the master realm via admin API with a known `clientId` + a known client secret → obtains a token via `grant_type=client_credentials` for that client → asserts the token's `azp` (authorized party) matches the new client's clientId → deletes the client (idempotent cleanup). | Proves the full lifecycle of admin-API client creation + service-account token issuance, the canonical "real app integrating with keycloak" flow. Non-vacuous: tests TWO grant types (password + client-credentials) and the admin-API CRUD on clients. |
|
||||
|
||||
Both tests run in the **custom** tier against the same `live_app` shared deployment as the
|
||||
lifecycle overlays — no extra deploy, no extra teardown.
|
||||
|
||||
@ -28,7 +28,7 @@ import urllib.parse
|
||||
import urllib.request
|
||||
import uuid
|
||||
|
||||
# kc_admin.py lives in tests/keycloak/, one level up from this file in tests/keycloak/functional/
|
||||
# kc_admin.py lives in tests/keycloak/, one level up from this file in tests/keycloak/custom/
|
||||
sys.path.insert(0, os.path.join(os.path.dirname(__file__), ".."))
|
||||
sys.path.insert(0, os.path.join(os.path.dirname(__file__), "..", "..", "..", "runner"))
|
||||
import kc_admin # noqa: E402
|
||||
@ -18,7 +18,7 @@ import os
|
||||
import sys
|
||||
import time
|
||||
|
||||
# kc_admin.py lives in tests/keycloak/, one level up from this file in tests/keycloak/functional/
|
||||
# kc_admin.py lives in tests/keycloak/, one level up from this file in tests/keycloak/custom/
|
||||
sys.path.insert(0, os.path.join(os.path.dirname(__file__), ".."))
|
||||
sys.path.insert(0, os.path.join(os.path.dirname(__file__), "..", "..", "..", "runner"))
|
||||
import kc_admin # noqa: E402
|
||||
@ -5,16 +5,16 @@ Phase-2 P2 mapping table. The Adversary cold-verifies parity by reading the sour
|
||||
|
||||
| recipe-maintainer file | cc-ci file | what's verified | status |
|
||||
|---|---|---|---|
|
||||
| `recipe-info/lasuite-docs/tests/health_check.py` | `tests/lasuite-docs/functional/test_health_check.py` | The app serves over HTTPS and returns a successful response (200/301/302). The cc-ci port preserves the assertion shape, adapted to the ephemeral per-run domain via `live_app`. | **ported** |
|
||||
| `recipe-info/lasuite-docs/tests/oidc_login.py` | `tests/lasuite-docs/functional/test_oidc_with_keycloak.py` (Q2.4 acceptance, partial port) + `test_auth_required.py` (proves the gate is wired) | The original's flow: deploy keycloak + setup realm/client/user + obtain JWT + use it against lasuite-docs's protected API. The cc-ci pair: (a) `test_oidc_with_keycloak` deploys keycloak as a Q2.3 dep, sets up realm/client/user, obtains a real JWT, validates iss/azp/typ/exp claims; (b) `test_auth_required` proves lasuite-docs's backend API requires auth (401). Step-(c) — actually USING the JWT against lasuite-docs — requires wiring the dep keycloak's client_secret + OIDC env into lasuite-docs's `.env` at install time; **see "Deferred (Q3.1 follow-up)" below**. | **partial; see follow-up** |
|
||||
| `recipe-info/lasuite-docs/tests/health_check.py` | `tests/lasuite-docs/custom/test_health_check.py` | The app serves over HTTPS and returns a successful response (200/301/302). The cc-ci port preserves the assertion shape, adapted to the ephemeral per-run domain via `live_app`. | **ported** |
|
||||
| `recipe-info/lasuite-docs/tests/oidc_login.py` | `tests/lasuite-docs/custom/test_oidc_with_keycloak.py` (Q2.4 acceptance, partial port) + `test_auth_required.py` (proves the gate is wired) | The original's flow: deploy keycloak + setup realm/client/user + obtain JWT + use it against lasuite-docs's protected API. The cc-ci pair: (a) `test_oidc_with_keycloak` deploys keycloak as a Q2.3 dep, sets up realm/client/user, obtains a real JWT, validates iss/azp/typ/exp claims; (b) `test_auth_required` proves lasuite-docs's backend API requires auth (401). Step-(c) — actually USING the JWT against lasuite-docs — requires wiring the dep keycloak's client_secret + OIDC env into lasuite-docs's `.env` at install time; **see "Deferred (Q3.1 follow-up)" below**. | **partial; see follow-up** |
|
||||
| `recipe-info/lasuite-docs/tests/upload_conversion.py` | (Q3.1 follow-up — needs OIDC env wired into lasuite-docs first) | The original uploads .md + .docx via authenticated `POST /api/v1.0/documents/<id>/upload` and asserts the y-provider + docspec conversion paths fire. The cc-ci port requires authentication, which requires OIDC env wiring (see below). | **deferred** |
|
||||
|
||||
## Recipe-specific tests (Phase-2 P3, ≥2 beyond parity)
|
||||
|
||||
| cc-ci file | what's verified | rationale |
|
||||
|---|---|---|
|
||||
| `tests/lasuite-docs/functional/test_oidc_with_keycloak.py` | Deploys keycloak as a per-run **dep** (Q2.3 resolver via `DEPS = ["keycloak"]`), sets up a realm/client/user, exercises the OIDC discovery endpoint + the password-grant flow against the dep keycloak, validates the returned JWT's iss/azp/typ/exp claims. | The recipe is **OIDC-dependent** by design; proving the SSO provider deploys + issues tokens + the JWT contract is intact is a defining lasuite-docs behavior (and the Q2 gate acceptance test). |
|
||||
| `tests/lasuite-docs/functional/test_auth_required.py` | GETs `/api/v1.0/users/me/` without a token; asserts **401 Unauthorized** (or 403). Non-vacuous: distinguishes a correctly-wired OIDC gate (401) from anonymous access (200), missing route (404), and broken backend (5xx). | Proves lasuite-docs's **own** auth posture (distinct from the SSO provider's token issuance). Together with `test_oidc_with_keycloak` this exercises both sides of the OIDC flow's plumbing. |
|
||||
| `tests/lasuite-docs/custom/test_oidc_with_keycloak.py` | Deploys keycloak as a per-run **dep** (Q2.3 resolver via `DEPS = ["keycloak"]`), sets up a realm/client/user, exercises the OIDC discovery endpoint + the password-grant flow against the dep keycloak, validates the returned JWT's iss/azp/typ/exp claims. | The recipe is **OIDC-dependent** by design; proving the SSO provider deploys + issues tokens + the JWT contract is intact is a defining lasuite-docs behavior (and the Q2 gate acceptance test). |
|
||||
| `tests/lasuite-docs/custom/test_auth_required.py` | GETs `/api/v1.0/users/me/` without a token; asserts **401 Unauthorized** (or 403). Non-vacuous: distinguishes a correctly-wired OIDC gate (401) from anonymous access (200), missing route (404), and broken backend (5xx). | Proves lasuite-docs's **own** auth posture (distinct from the SSO provider's token issuance). Together with `test_oidc_with_keycloak` this exercises both sides of the OIDC flow's plumbing. |
|
||||
|
||||
Two specific tests — the ≥2 floor is met. Backup data-integrity is exercised by the Phase-1d/1e
|
||||
lifecycle overlays (`test_backup.py`/`test_restore.py` + `ops.py`).
|
||||
@ -30,7 +30,7 @@ with a real OIDC-issued JWT. To round these out, the cc-ci side needs:
|
||||
- Inserts `SECRET_OIDC_RPCS_VERSION=v1` + the secret value via `abra app secret insert`.
|
||||
- Appends to lasuite-docs's `.env`: `OIDC_REALM`, `OIDC_CLIENT_ID`, `OIDC_OP_*` URLs pointing
|
||||
at the dep keycloak.
|
||||
2. **Authenticated test**: a new `tests/lasuite-docs/functional/test_create_doc.py` performs the
|
||||
2. **Authenticated test**: a new `tests/lasuite-docs/custom/test_create_doc.py` performs the
|
||||
password grant against the dep keycloak, presents the JWT to lasuite-docs's
|
||||
`POST /api/v1.0/documents/` (create a doc), asserts the doc is fetched back via
|
||||
`GET /api/v1.0/documents/<id>/` — the §4.3 prescribed create-and-read-back.
|
||||
|
||||
@ -1,6 +1,6 @@
|
||||
#!/usr/bin/env bash
|
||||
# lasuite-docs — INSTALL-TIME OIDC wiring hook (rcust P2b; migrated from the deleted
|
||||
# setup_custom_tests.sh post-deploy path — sibling of lasuite-drive/-meet's hooks).
|
||||
# old post-deploy setup path — sibling of lasuite-drive/-meet's hooks).
|
||||
#
|
||||
# Runs during the install tier AFTER `abra app new` + EXTRA_ENV + `abra app secret generate`, and
|
||||
# BEFORE the single `abra app deploy` (lifecycle.py::_run_install_steps). Writing OIDC env + the
|
||||
|
||||
@ -10,7 +10,7 @@ HTTP_TIMEOUT = 600
|
||||
|
||||
# Phase 2 Q2.3 deps: lasuite-docs's recipe-maintainer corpus declares `requires = ["keycloak"]`.
|
||||
# Declaring it here makes the orchestrator deploy a per-run keycloak BEFORE lasuite-docs so the
|
||||
# OIDC-flow functional test (`functional/test_oidc_with_keycloak.py`) can run against a real
|
||||
# OIDC-flow custom test (`custom/test_oidc_with_keycloak.py`) can run against a real
|
||||
# provider in the same run. The dep is undeployed AFTER the parent in the orchestrator's `finally`.
|
||||
DEPS = ["keycloak"]
|
||||
|
||||
|
||||
@ -5,16 +5,16 @@ Phase-2 P2 mapping table. The Adversary cold-verifies parity by reading the sour
|
||||
|
||||
**Enrollment status:** Q3.2 SSO iteration. Base deploy + lifecycle (install/upgrade/backup/restore
|
||||
data-integrity) + parity health_check landed first; the base proved cold-green @2026-05-28 (all 12
|
||||
services incl. onlyoffice+collabora). Now landed on top: `DEPS=["keycloak"]` + `setup_custom_tests.sh`
|
||||
OIDC wiring + the OIDC SSO test + the MinIO storage round-trip (the §4.3 specifics). WOPI discovery is
|
||||
services incl. onlyoffice+collabora). Now landed on top: `DEPS=["keycloak"]` + install-time OIDC
|
||||
wiring + the OIDC SSO test + the MinIO storage round-trip (the §4.3 specifics). WOPI discovery is
|
||||
a further (3rd) test beyond the ≥2 floor — still planned. This file is updated as each row lands;
|
||||
nothing is a silent omission.
|
||||
|
||||
| recipe-maintainer file | cc-ci file | what's verified | status |
|
||||
|---|---|---|---|
|
||||
| `recipe-info/lasuite-drive/tests/health_check.py` | `tests/lasuite-drive/functional/test_health_check.py` | App serves over HTTPS and returns 200/301/302 from `/`. Port preserves the assertion shape, adapted to the ephemeral per-run domain via `live_app`. | **ported** |
|
||||
| `recipe-info/lasuite-drive/tests/oidc_login.py` | `tests/lasuite-drive/functional/test_oidc_with_keycloak.py` | Original: Drive `/api/v1.0/authenticate/` redirects to Keycloak → password-grant token → `/api/v1.0/users/me/` returns the user. cc-ci port deploys keycloak as a per-run dep (`DEPS=["keycloak"]`), wires OIDC env via `setup_custom_tests.sh`, exercises discovery + password grant + JWT claims (iss/azp/typ/exp) against the dep realm `lasuite-drive` (mirrors the proven lasuite-docs `test_oidc_with_keycloak`). `@requires_deps` so a deps-not-ready skip fails the run (F2-11), not a silent green. | **ported** |
|
||||
| `recipe-info/lasuite-drive/tests/wopi_configured.py` | `tests/lasuite-drive/functional/test_wopi_configured.py` (planned) | Original: Collabora + OnlyOffice WOPI discovery endpoints return valid WOPI XML. cc-ci port checks the Collabora discovery XML over the flattened `collabora-<domain>` route (pure HTTP, no browser/SSO). | **pending** |
|
||||
| `recipe-info/lasuite-drive/tests/health_check.py` | `tests/lasuite-drive/custom/test_health_check.py` | App serves over HTTPS and returns 200/301/302 from `/`. Port preserves the assertion shape, adapted to the ephemeral per-run domain via `live_app`. | **ported** |
|
||||
| `recipe-info/lasuite-drive/tests/oidc_login.py` | `tests/lasuite-drive/custom/test_oidc_with_keycloak.py` | Original: Drive `/api/v1.0/authenticate/` redirects to Keycloak → password-grant token → `/api/v1.0/users/me/` returns the user. cc-ci port deploys keycloak as a per-run dep (`DEPS=["keycloak"]`), wires OIDC env at install time, exercises discovery + password grant + JWT claims (iss/azp/typ/exp) against the dep realm `lasuite-drive` (mirrors the proven lasuite-docs `test_oidc_with_keycloak`). `@requires_deps` so a deps-not-ready skip fails the run (F2-11), not a silent green. | **ported** |
|
||||
| `recipe-info/lasuite-drive/tests/wopi_configured.py` | `tests/lasuite-drive/custom/test_wopi_configured.py` (planned) | Original: Collabora + OnlyOffice WOPI discovery endpoints return valid WOPI XML. cc-ci port checks the Collabora discovery XML over the flattened `collabora-<domain>` route (pure HTTP, no browser/SSO). | **pending** |
|
||||
| `recipe-info/lasuite-drive/tests/wopi_on_startup.py` | (see DECISIONS / DEFERRED) | Original: greps celery worker container logs for the entrypoint WOPI trigger. cc-ci port via `docker service logs` on the celery service. | **pending** |
|
||||
| `recipe-info/lasuite-drive/tests/celery_beat_wopi.py` | (likely DEFERRED — "thorough mode only") | Original sleeps 15–90s waiting for Celery Beat to fire; recipe-maintainer marks it "thorough mode only". Candidate for the `--extra-tests` opt-in (DEFERRED.md), like the matrix-synapse operational ports. | **likely deferred** |
|
||||
|
||||
@ -22,9 +22,9 @@ nothing is a silent omission.
|
||||
|
||||
| cc-ci file | what's verified | status |
|
||||
|---|---|---|
|
||||
| `functional/test_oidc_with_keycloak.py` | SSO round-trip against the dep keycloak: OIDC discovery advertises realm `lasuite-drive`; password grant yields a valid JWT with iss/azp/typ/exp claims. Drive is OIDC-required — this is its defining auth path. | **landed** |
|
||||
| `functional/test_minio_storage.py` | The §4.3 create-an-object + read-it-back, at Drive's storage layer: confirms the `drive-media-storage` MinIO bucket exists, then a real upload → list → download round-trip (unique marker) asserting the bytes survive. Runs `mc` inside the `minio` container with the in-container root creds. Non-health-only: a missing bucket or broken object store fails it. | **landed** |
|
||||
| `functional/test_wopi_configured.py` (planned, 3rd beyond floor) | Collabora WOPI discovery XML served + valid over the flattened `collabora-<domain>` route — Drive's in-browser office-editing feature. | **planned** |
|
||||
| `custom/test_oidc_with_keycloak.py` | SSO round-trip against the dep keycloak: OIDC discovery advertises realm `lasuite-drive`; password grant yields a valid JWT with iss/azp/typ/exp claims. Drive is OIDC-required — this is its defining auth path. | **landed** |
|
||||
| `custom/test_minio_storage.py` | The §4.3 create-an-object + read-it-back, at Drive's storage layer: confirms the `drive-media-storage` MinIO bucket exists, then a real upload → list → download round-trip (unique marker) asserting the bytes survive. Runs `mc` inside the `minio` container with the in-container root creds. Non-health-only: a missing bucket or broken object store fails it. | **landed** |
|
||||
| `custom/test_wopi_configured.py` (planned, 3rd beyond floor) | Collabora WOPI discovery XML served + valid over the flattened `collabora-<domain>` route — Drive's in-browser office-editing feature. | **planned** |
|
||||
|
||||
## Backup data-integrity (P4) — landed
|
||||
|
||||
|
||||
@ -12,7 +12,7 @@ Mirrors the proven lasuite-docs SSO model:
|
||||
`deps-not-ready` reason — and (per F2-11) the orchestrator then fails the run rather than going
|
||||
green on a skipped SSO test.
|
||||
|
||||
SOURCE: adapted from tests/lasuite-docs/functional/test_oidc_with_keycloak.py (Q2.4 acceptance).
|
||||
SOURCE: adapted from tests/lasuite-docs/custom/test_oidc_with_keycloak.py (Q2.4 acceptance).
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
@ -14,14 +14,14 @@ from harness import lifecycle # noqa: E402
|
||||
|
||||
|
||||
def pre_install(ctx):
|
||||
"""Post-deploy seed for the custom tier (the former setup_custom_tests.sh, moved here in rcust
|
||||
"""Post-deploy seed for the custom tier (the former post-deploy setup hook, moved here in rcust
|
||||
P2b — install_steps.sh runs PRE-deploy and cannot touch the live stack). The deploy alone does
|
||||
NOT create the MinIO bucket: `minio-createbuckets` is a `replicas:0` one-shot (restart_policy:
|
||||
none) that must be triggered. The MinIO storage test asserts the bucket exists, so trigger it
|
||||
here and poll. `--detach` is REQUIRED: the job creates the bucket then EXITS 0, so it never
|
||||
holds a steady 1/1 replica — a blocking scale would wait forever.
|
||||
|
||||
BEST-EFFORT, like the setup_custom_tests.sh it replaced: on poll timeout we WARN and continue
|
||||
BEST-EFFORT, like the old post-deploy hook it replaced: on poll timeout we WARN and continue
|
||||
(the one-shot often lands just after the window). The custom-tier MinIO storage test is the
|
||||
real gate for a genuinely missing bucket — failing the install op here was an rcust M2
|
||||
regression (the original hook fell through on timeout by design)."""
|
||||
|
||||
@ -20,7 +20,7 @@ HTTP_TIMEOUT = 900
|
||||
# Base deploy/lifecycle proven cold-green @2026-05-28 (install: pass; 12 services incl.
|
||||
# onlyoffice+collabora) once the Docker Hub rate limit was fixed. Declaring DEPS makes the
|
||||
# orchestrator provision keycloak (realm/client/user) BEFORE the single deploy;
|
||||
# functional/test_oidc_with_keycloak.py then exercises the SSO flow.
|
||||
# custom/test_oidc_with_keycloak.py then exercises the SSO flow.
|
||||
DEPS = ["keycloak"]
|
||||
|
||||
# OIDC is wired at INSTALL time (the only deps mode since rcust P2b; Q3.2a pioneered it here):
|
||||
@ -28,7 +28,7 @@ DEPS = ["keycloak"]
|
||||
# `abra app deploy`, and tests/lasuite-drive/install_steps.sh writes the OIDC env + client secret
|
||||
# into the .env that one deploy reads. No post-deploy reconverge (the flaky 12-service collabora
|
||||
# WOPI race is structurally gone). The post-deploy MinIO bucket one-shot lives in ops.py
|
||||
# pre_install (the former setup_custom_tests.sh, deleted in P2b).
|
||||
# pre_install (the former post-deploy setup hook, moved here in P2b).
|
||||
|
||||
|
||||
def READY_PROBE(ctx):
|
||||
|
||||
@ -6,9 +6,9 @@ Reference corpus: `references/recipe-maintainer/recipe-info/lasuite-meet/tests/`
|
||||
|
||||
| recipe-maintainer test | cc-ci test | what's verified (same thing) |
|
||||
|---|---|---|
|
||||
| `health_check.py` | `tests/lasuite-meet/functional/test_health_check.py::test_lasuite_meet_returns_200` | HTTP 200/301/302 from `/` (SPA shell served). |
|
||||
| `oidc_login.py` | `tests/lasuite-meet/functional/test_oidc_with_keycloak.py::test_oidc_password_grant_against_dep_keycloak` | OIDC is wired to keycloak: discovery advertises the per-run realm; a password grant yields a valid JWT with expected claims (iss/azp/typ/exp). Meet is OIDC-REQUIRED; OIDC wired at install (install_steps.sh, OIDC_AT_INSTALL). |
|
||||
| `meeting_flow.py` | `tests/lasuite-meet/functional/test_meeting_flow.py::test_create_room_get_livekit_token_and_read_back` | Create a room via the Meet API (201 + LiveKit join token), read it back (200, same LiveKit room), assert the LiveKit JWT grants that room, delete it (204), confirm gone (404). |
|
||||
| `health_check.py` | `tests/lasuite-meet/custom/test_health_check.py::test_lasuite_meet_returns_200` | HTTP 200/301/302 from `/` (SPA shell served). |
|
||||
| `oidc_login.py` | `tests/lasuite-meet/custom/test_oidc_with_keycloak.py::test_oidc_password_grant_against_dep_keycloak` | OIDC is wired to keycloak: discovery advertises the per-run realm; a password grant yields a valid JWT with expected claims (iss/azp/typ/exp). Meet is OIDC-REQUIRED; OIDC wired at install (install_steps.sh, OIDC_AT_INSTALL). |
|
||||
| `meeting_flow.py` | `tests/lasuite-meet/custom/test_meeting_flow.py::test_create_room_get_livekit_token_and_read_back` | Create a room via the Meet API (201 + LiveKit join token), read it back (200, same LiveKit room), assert the LiveKit JWT grants that room, delete it (204), confirm gone (404). |
|
||||
|
||||
## Recipe-specific functional tests (P3, ≥2 beyond bare parity)
|
||||
1. **`test_meeting_flow.py`** — §4.3 create-an-object + read-it-back: create a room → GET it back →
|
||||
|
||||
@ -12,7 +12,7 @@ Mirrors the proven lasuite-docs SSO model:
|
||||
`deps-not-ready` reason — and (per F2-11) the orchestrator then fails the run rather than going
|
||||
green on a skipped SSO test.
|
||||
|
||||
SOURCE: adapted from tests/lasuite-docs/functional/test_oidc_with_keycloak.py (Q2.4 acceptance).
|
||||
SOURCE: adapted from tests/lasuite-docs/custom/test_oidc_with_keycloak.py (Q2.4 acceptance).
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
@ -16,10 +16,10 @@ email stack: nginx front + admin + postfix/smtp + dovecot/imap + rspamd/antispam
|
||||
(cc-ci-run) tests reach SMTP/IMAP at 127.0.0.1.
|
||||
|
||||
## Recipe-specific functional tests (P3 — ≥2)
|
||||
1. `functional/test_mailbox.py` — §4.3 create-an-object + read-back: create a mailbox via the admin
|
||||
1. `custom/test_mailbox.py` — §4.3 create-an-object + read-back: create a mailbox via the admin
|
||||
container's `flask mailu user` CLI, then read it back from `flask mailu config-export --json` and
|
||||
assert the address is present (admin-DB provisioning round-trip).
|
||||
2. `functional/test_mail_flow.py` — the characteristic end-to-end mail flow: INJECT a uniquely-marked
|
||||
2. `custom/test_mail_flow.py` — the characteristic end-to-end mail flow: INJECT a uniquely-marked
|
||||
message to the mailbox via the postfix container's local `sendmail` (locally-originated → not
|
||||
greylisted), then VERIFY delivery+storage via dovecot's `doveadm search` in the imap container —
|
||||
a real postfix → rspamd → dovecot deliver/store/fetch round-trip. We use the in-container mail
|
||||
|
||||
@ -11,7 +11,7 @@ import time
|
||||
sys.path.insert(0, os.path.join(os.path.dirname(__file__), "..", "..", "runner"))
|
||||
from harness import lifecycle # noqa: E402
|
||||
|
||||
sys.path.insert(0, os.path.join(os.path.dirname(__file__), "functional"))
|
||||
sys.path.insert(0, os.path.join(os.path.dirname(__file__), "custom"))
|
||||
import _mailu # noqa: E402
|
||||
|
||||
_CI_LOCALPART = "citest"
|
||||
|
||||
@ -10,7 +10,7 @@ from __future__ import annotations
|
||||
import os
|
||||
import sys
|
||||
|
||||
sys.path.insert(0, os.path.join(os.path.dirname(__file__), "functional"))
|
||||
sys.path.insert(0, os.path.join(os.path.dirname(__file__), "custom"))
|
||||
import _mailu # noqa: E402
|
||||
|
||||
sys.path.insert(0, os.path.join(os.path.dirname(__file__), "..", "..", "runner"))
|
||||
|
||||
@ -10,7 +10,7 @@ from __future__ import annotations
|
||||
import os
|
||||
import sys
|
||||
|
||||
sys.path.insert(0, os.path.join(os.path.dirname(__file__), "functional"))
|
||||
sys.path.insert(0, os.path.join(os.path.dirname(__file__), "custom"))
|
||||
import _mailu # noqa: E402
|
||||
|
||||
sys.path.insert(0, os.path.join(os.path.dirname(__file__), "..", "..", "runner"))
|
||||
|
||||
@ -8,7 +8,7 @@ messages, federation) in Python tests adapted to the ephemeral per-run-deploy mo
|
||||
|
||||
| recipe-maintainer file | cc-ci file | what's verified | status |
|
||||
|---|---|---|---|
|
||||
| (no health_check.py in the recipe-maintainer corpus) | `tests/matrix-synapse/functional/test_health_check.py` | HTTP 200 + JSON document from `/_matrix/client/versions` (the synapse client API). | **Phase-2 health_check** (aligned with the parity-port convention; the corpus has no health_check.py to port from). |
|
||||
| (no health_check.py in the recipe-maintainer corpus) | `tests/matrix-synapse/custom/test_health_check.py` | HTTP 200 + JSON document from `/_matrix/client/versions` (the synapse client API). | **Phase-2 health_check** (aligned with the parity-port convention; the corpus has no health_check.py to port from). |
|
||||
| `recipe-info/matrix-synapse/tests/compress_state.sh` | (deferred to Q4 follow-up — synapse_auto_compressor + state-group bloat) | The original creates state groups WITHOUT edges (full snapshots — Synapse's bloat pattern), runs the synapse_auto_compressor, asserts row counts drop. Requires per-run admin user pre-seeded + a long-running synapse + access to the synapse_auto_compressor binary. | **deferred** (operational complexity; needs custom install_steps.sh + admin user pre-seeding) |
|
||||
| `recipe-info/matrix-synapse/tests/test_complexity_limit.sh` | (deferred to Q4 follow-up — rate-limit behaviour) | Exercises Synapse's complexity-limit rejection of huge events. | **deferred** (load-test class; needs many-event setup) |
|
||||
| `recipe-info/matrix-synapse/tests/test_purge.sh` | (deferred to Q4 follow-up — admin purge commands) | Tests the abra.sh `db purge_history`, `db purge_room` etc. helpers. Operational tests against the recipe's helper shell wrappers. | **deferred** (recipe-helper-script tests, not synapse-behavior tests; orthogonal to Phase-2 P3) |
|
||||
@ -28,8 +28,8 @@ Three specific tests landed (beyond parity health_check):
|
||||
|
||||
| cc-ci file | what's verified | rationale |
|
||||
|---|---|---|
|
||||
| `tests/matrix-synapse/functional/test_federation_version.py` | GET `/_matrix/federation/v1/version` → 200, JSON with `server.name == "Synapse"`, non-empty `server.version`. | Plan §4.3 prescribed. Federation discovery endpoint — the recipe's "is this a real Synapse, federation-ready" surface. Non-vacuous: a Dendrite or misconfigured federation subsystem fails. |
|
||||
| `tests/matrix-synapse/functional/test_register_and_message.py` | **Plan §4.3 prescribed create-and-read-back.** Reads the abra-generated `registration` shared secret from the synapse container; registers two users (alice + bob) via `/_synapse/admin/v1/register` (HMAC-SHA1 nonce flow); both login via `/_matrix/client/v3/login`; alice creates a private_chat room; invites bob; bob joins; alice sends a uniquely-marked m.room.message; bob reads the room's messages and finds the marker. | The canonical Matrix create-and-read-back, exercising registration + login + room create/invite/join + send/receive across the full client API. Non-vacuous: each step fails at the level it's broken (admin API, login, room ops, send/receive); marker assertion confirms the message actually round-tripped across two users. |
|
||||
| `tests/matrix-synapse/custom/test_federation_version.py` | GET `/_matrix/federation/v1/version` → 200, JSON with `server.name == "Synapse"`, non-empty `server.version`. | Plan §4.3 prescribed. Federation discovery endpoint — the recipe's "is this a real Synapse, federation-ready" surface. Non-vacuous: a Dendrite or misconfigured federation subsystem fails. |
|
||||
| `tests/matrix-synapse/custom/test_register_and_message.py` | **Plan §4.3 prescribed create-and-read-back.** Reads the abra-generated `registration` shared secret from the synapse container; registers two users (alice + bob) via `/_synapse/admin/v1/register` (HMAC-SHA1 nonce flow); both login via `/_matrix/client/v3/login`; alice creates a private_chat room; invites bob; bob joins; alice sends a uniquely-marked m.room.message; bob reads the room's messages and finds the marker. | The canonical Matrix create-and-read-back, exercising registration + login + room create/invite/join + send/receive across the full client API. Non-vacuous: each step fails at the level it's broken (admin API, login, room ops, send/receive); marker assertion confirms the message actually round-tripped across two users. |
|
||||
|
||||
Media upload/download deferred — would add a fourth specific test (`media_upload_roundtrip`)
|
||||
using `/_matrix/media/v3/upload` + `/_matrix/media/v3/download/<server>/<media_id>`. Not in this
|
||||
|
||||
@ -12,14 +12,14 @@ tiers run by default (the Phase-1e invariant: no overlay ⇒ generic runs). The
|
||||
postgres in-compose (no external dep), so no dependency resolution is needed.
|
||||
|
||||
## P3 — Recipe-specific functional tests (≥2 separate characteristic tests)
|
||||
1. `functional/test_create_message.py::test_create_message_roundtrip` — **§4.3 create-an-object +
|
||||
1. `custom/test_create_message.py::test_create_message_roundtrip` — **§4.3 create-an-object +
|
||||
read-it-back**: first user (system admin) → login → create team → create channel → POST a unique
|
||||
marker message → GET it back by id → assert the text round-trips.
|
||||
2. `functional/test_multiuser_message.py::test_second_user_reads_first_users_message` — the defining
|
||||
2. `custom/test_multiuser_message.py::test_second_user_reads_first_users_message` — the defining
|
||||
**team-chat** behaviour (distinct code path: membership + ACL + cross-user delivery): user_a posts a
|
||||
unique marker; a SECOND user (created via admin API, added to team+channel) logs in with its own
|
||||
session and GETs the channel posts → asserts it sees user_a's message. Not a self read-back.
|
||||
- `functional/test_health_check.py` — `test_root_serves` (`/` 200/302) + `test_system_ping_ok`
|
||||
- `custom/test_health_check.py` — `test_root_serves` (`/` 200/302) + `test_system_ping_ok`
|
||||
(`/api/v4/system/ping` → `{"status":"OK"}`, API liveness). Supporting health/liveness, not counted
|
||||
toward the P3 ≥2 floor.
|
||||
|
||||
|
||||
@ -12,9 +12,9 @@ publishes 64738 on the cc-ci host so the on-host (cc-ci-run) protocol tests conn
|
||||
|
||||
| recipe-maintainer test (`recipe-info/mumble/tests/`) | what it verifies | cc-ci test | same thing? |
|
||||
|---|---|---|---|
|
||||
| `health_check.py` | mumble server listening on TCP 64738 | `functional/test_tcp_health.py` | yes — TCP connect to 64738 (host-published) |
|
||||
| `mumble_connect.py` | full TLS protocol handshake: TLS connect, server Version, auth accepted (no Reject), channel list present, ServerSync handshake completes, welcome text | `functional/test_protocol_handshake.py` (+ `functional/_mumble_proto.py`, adapted from the corpus's stdlib protobuf/protocol code) | yes — same handshake; asserts tls_connect + version + auth_accepted + channel presence + ServerSync |
|
||||
| `web_client.py` | mumble-web client reachable over HTTPS, HTTP 200, page contains `Mumble` + `config.js`, valid HTML | `functional/test_web_client.py` | yes — same 200 + body markers (`Mumble`, `config.js`, `<!DOCTYPE html>`) |
|
||||
| `health_check.py` | mumble server listening on TCP 64738 | `custom/test_tcp_health.py` | yes — TCP connect to 64738 (host-published) |
|
||||
| `mumble_connect.py` | full TLS protocol handshake: TLS connect, server Version, auth accepted (no Reject), channel list present, ServerSync handshake completes, welcome text | `custom/test_protocol_handshake.py` (+ `custom/_mumble_proto.py`, adapted from the corpus's stdlib protobuf/protocol code) | yes — same handshake; asserts tls_connect + version + auth_accepted + channel presence + ServerSync |
|
||||
| `web_client.py` | mumble-web client reachable over HTTPS, HTTP 200, page contains `Mumble` + `config.js`, valid HTML | `custom/test_web_client.py` | yes — same 200 + body markers (`Mumble`, `config.js`, `<!DOCTYPE html>`) |
|
||||
|
||||
No recipe-maintainer mumble test is omitted — all three are ported. No `DECISIONS.md` non-port
|
||||
entry is needed for mumble.
|
||||
@ -27,10 +27,10 @@ prove our deploy-time configuration propagated into the running murmur server an
|
||||
delivered over the real protocol (version-independent — they assert OUR configured markers, not
|
||||
hard-coded upstream values):
|
||||
|
||||
1. `functional/test_welcome_text_roundtrip.py` — deploys with a unique `WELCOME_TEXT` marker
|
||||
1. `custom/test_welcome_text_roundtrip.py` — deploys with a unique `WELCOME_TEXT` marker
|
||||
(`recipe_meta.EXTRA_ENV` → `MUMBLE_CONFIG_WELCOMETEXT`); asserts that exact marker surfaces in the
|
||||
server's ServerSync `welcome_text` delivered to a connecting client. (create config → read back.)
|
||||
2. `functional/test_server_config_limits.py` — deploys with a distinctive non-default `USERS=42`
|
||||
2. `custom/test_server_config_limits.py` — deploys with a distinctive non-default `USERS=42`
|
||||
(max-users cap → `MUMBLE_CONFIG_USERS`); asserts the server's ServerConfig message reports
|
||||
`max_users == 42` (and a well-formed `allow_html`), proving the recipe wires deploy-time
|
||||
server-capacity policy into the running server.
|
||||
|
||||
@ -1,13 +1,13 @@
|
||||
# Parity — n8n
|
||||
|
||||
Phase-2 P2 mapping table: every `references/recipe-maintainer/recipe-info/n8n/tests/*.py` has a
|
||||
comparable cc-ci test under `tests/n8n/functional/`, asserting the **same thing** (not a renamed
|
||||
comparable cc-ci test under `tests/n8n/custom/`, asserting the **same thing** (not a renamed
|
||||
file). The Adversary cold-verifies parity by reading the source `recipe-info/<file>` and the cc-ci
|
||||
file side-by-side.
|
||||
|
||||
| recipe-maintainer file | cc-ci file | what's verified | status |
|
||||
|---|---|---|---|
|
||||
| `recipe-info/n8n/tests/health_check.py` | `tests/n8n/functional/test_health_check.py` | The app is reachable over HTTPS and returns a successful response (the original asserted HTTP 200 against a persistent `n8n.<suffix>` host). The cc-ci port preserves the assertion shape — HTTP 200 from the served root — and adapts to the ephemeral per-run domain via the `live_app` fixture. | **ported** |
|
||||
| `recipe-info/n8n/tests/health_check.py` | `tests/n8n/custom/test_health_check.py` | The app is reachable over HTTPS and returns a successful response (the original asserted HTTP 200 against a persistent `n8n.<suffix>` host). The cc-ci port preserves the assertion shape — HTTP 200 from the served root — and adapts to the ephemeral per-run domain via the `live_app` fixture. | **ported** |
|
||||
|
||||
## Recipe-specific tests (Phase-2 P3 §4.3 floor: "create-an-object + read-it-back, and one more")
|
||||
|
||||
@ -16,9 +16,9 @@ directly: "create a workflow via API, execute it, assert the result." So:
|
||||
|
||||
| cc-ci file | what's verified | rationale |
|
||||
|---|---|---|
|
||||
| `tests/n8n/functional/test_workflow_roundtrip.py` | Owner setup via `POST /rest/owner/setup` with a per-run-generated email + password (class-B run-scoped secret, plan §4.4-B); then `POST /rest/workflows` creates a Manual-Trigger workflow with a unique name; then `GET /rest/workflows/<id>` reads it back; asserts the returned id matches, name matches, nodes payload preserved (type/name of the one node). | **Plan §4.3 prescribed test** — create-an-object + read-it-back, exercising n8n's persistence + retrieval. Non-vacuous: a broken persistence layer would round-trip with wrong shape; a wedged engine that serves the SPA but rejects workflow POSTs fails at the create step. |
|
||||
| `tests/n8n/functional/test_rest_settings.py` | Polls `/rest/settings` until response is **application/json** (rejects the "n8n is starting up" SPA placeholder); asserts known public-settings keys (`userManagement`, `defaultLocale`, `authCookie`) in the `data` envelope. | The editor SPA's primary API contract — proves bootstrap surface is intact. Distinct from `test_workflow_roundtrip.py` (which proves persistence); this proves the SPA can come up at all. |
|
||||
| `tests/n8n/functional/test_login_state.py` | Polls `/rest/login` until response is **application/json**; asserts JSON dict/list shape — proves the user-management/auth subsystem initialized. | Auth subsystem readiness; distinct from settings (a broken auth backend would let settings return JSON but login would 5xx). |
|
||||
| `tests/n8n/custom/test_workflow_roundtrip.py` | Owner setup via `POST /rest/owner/setup` with a per-run-generated email + password (class-B run-scoped secret, plan §4.4-B); then `POST /rest/workflows` creates a Manual-Trigger workflow with a unique name; then `GET /rest/workflows/<id>` reads it back; asserts the returned id matches, name matches, nodes payload preserved (type/name of the one node). | **Plan §4.3 prescribed test** — create-an-object + read-it-back, exercising n8n's persistence + retrieval. Non-vacuous: a broken persistence layer would round-trip with wrong shape; a wedged engine that serves the SPA but rejects workflow POSTs fails at the create step. |
|
||||
| `tests/n8n/custom/test_rest_settings.py` | Polls `/rest/settings` until response is **application/json** (rejects the "n8n is starting up" SPA placeholder); asserts known public-settings keys (`userManagement`, `defaultLocale`, `authCookie`) in the `data` envelope. | The editor SPA's primary API contract — proves bootstrap surface is intact. Distinct from `test_workflow_roundtrip.py` (which proves persistence); this proves the SPA can come up at all. |
|
||||
| `tests/n8n/custom/test_login_state.py` | Polls `/rest/login` until response is **application/json**; asserts JSON dict/list shape — proves the user-management/auth subsystem initialized. | Auth subsystem readiness; distinct from settings (a broken auth backend would let settings return JSON but login would 5xx). |
|
||||
|
||||
Three specific tests, exceeding the ≥2 floor — `test_workflow_roundtrip.py` is the plan §4.3
|
||||
prescribed "create + read-back"; the other two are bootstrap-readiness assertions retained from
|
||||
|
||||
@ -20,10 +20,10 @@ a true readiness gate. `DEPLOY_TIMEOUT` / `HTTP_TIMEOUT` are widened to 1200s to
|
||||
ClickHouse + migrations init.
|
||||
|
||||
## P3 — Recipe-specific functional tests
|
||||
- `functional/test_health_check.py`
|
||||
- `custom/test_health_check.py`
|
||||
- `test_plausible_root_serves` — GET `/api/health` → 200, proving ClickHouse + postgres + the
|
||||
sites_cache are all up (plausible's self-reported backend readiness; not a Traefik fallback).
|
||||
- `functional/test_event_tracking.py` — **§4.3 prescribed "track a test event, query it back"**, the
|
||||
- `custom/test_event_tracking.py` — **§4.3 prescribed "track a test event, query it back"**, the
|
||||
app's primary object. Both tests register a site row in the metadata postgres (plausible's
|
||||
`sites_cache` drops events for unregistered domains — empirically confirmed), POST to the public
|
||||
`/api/event` ingestion endpoint with a browser User-Agent (plausible drops bot/library UAs), then
|
||||
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user