docs(1d): DG8 — docs/testing.md (generic suite + overlay convention + install-steps hook); update enroll-recipe.md to the deploy-once contract; README pointer
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
@ -31,6 +31,8 @@ reference (`nixos-rebuild switch --flake '…#cc-ci'`) is unchanged.
|
||||
## Docs
|
||||
|
||||
- `docs/install.md` — rebuild the server from scratch (D8)
|
||||
- `docs/testing.md` — test architecture: generic lifecycle suite + layered recipe overlays
|
||||
(override/extend, discovery precedence, custom install-steps hook)
|
||||
- `docs/enroll-recipe.md` — add a recipe under CI (D5)
|
||||
- `docs/secrets.md` — secret model + rotation (D6)
|
||||
- `docs/architecture.md`, `docs/runbook.md` — design + debugging failed runs
|
||||
|
||||
@ -14,26 +14,40 @@ those are discovered and run against the live app (D4 — see below).
|
||||
```
|
||||
tests/<recipe>/
|
||||
├── recipe_meta.py # optional per-recipe harness config (see below)
|
||||
├── test_install.py # install-stage assertions (health + Playwright)
|
||||
├── test_upgrade.py # upgrade-stage assertions (data survives)
|
||||
└── test_backup.py # backup→mutate→restore assertions
|
||||
├── install_steps.sh # optional custom install-steps hook (pre-deploy setup)
|
||||
├── test_install.py # optional install overlay (else the generic install tier runs)
|
||||
├── test_upgrade.py # optional upgrade overlay (else the generic upgrade tier runs)
|
||||
├── test_backup.py # optional backup overlay (else the generic backup tier runs)
|
||||
└── test_restore.py # optional restore overlay (else the generic restore tier runs)
|
||||
```
|
||||
|
||||
Copy from an existing recipe (e.g. `tests/custom-html/` for a simple app, `tests/keycloak/` for a
|
||||
DB-backed one). The shared fixtures live in `tests/conftest.py` + `runner/harness/` — **do not edit
|
||||
them to add a recipe**; instead set per-recipe config in `recipe_meta.py`:
|
||||
**A recipe is testable with ZERO config:** with no overlay files, the **generic lifecycle suite**
|
||||
runs (install/upgrade/backup/restore) against a single shared deployment — see `docs/testing.md` for
|
||||
the full model (tiers, deploy-once, override-vs-extend, precedence, the install-steps hook). The
|
||||
per-recipe dir only holds the bits where the recipe needs *more* than the generic.
|
||||
|
||||
To add recipe-specific coverage, drop a `tests/<recipe>/test_<op>.py` **overlay** (it OVERRIDES the
|
||||
generic for that op; absent ⇒ generic runs). Overlays are **assertion-only** against the shared live
|
||||
deployment (the `live_app` fixture; they never deploy), and reuse the generic op + serving check by
|
||||
composition (`from harness import generic; generic.do_upgrade(...)` etc.), adding recipe-specific
|
||||
assertions. Copy an existing overlay (`tests/custom-html/` simple/volume marker; `tests/keycloak/`
|
||||
admin-API; `tests/matrix-synapse/` `db`-service psql marker). **Do not edit the shared
|
||||
`tests/conftest.py` / `runner/harness/` to add a recipe** — set per-recipe config in `recipe_meta.py`:
|
||||
|
||||
```python
|
||||
HEALTH_PATH = "/realms/master" # path that returns a healthy status (default "/")
|
||||
HEALTH_OK = (200,) # acceptable status codes (default 200/301/302)
|
||||
DEPLOY_TIMEOUT = 600 # seconds for services to converge (default 600)
|
||||
HTTP_TIMEOUT = 600 # seconds for the app to answer (default 300)
|
||||
BACKUP_CAPABLE = True # override backup-capability auto-detect (default: scan compose)
|
||||
EXTRA_ENV = {"KEY": "value"} # or EXTRA_ENV(domain) -> dict; extra .env keys set at deploy
|
||||
```
|
||||
|
||||
The test files use the fixtures: `deployed_app` (install), `deployed` (function-scoped), and the
|
||||
`harness.lifecycle` helpers (`http_get`, `http_body`, `exec_in_app`, `upgrade_app`, `backup_app`,
|
||||
`restore_app`, `previous_version`). The harness forces `LETS_ENCRYPT_ENV=""` (no ACME) and a unique
|
||||
short domain per run, and guarantees teardown.
|
||||
Useful `harness.lifecycle` helpers for overlays: `http_get`, `http_fetch`, `http_body`,
|
||||
`exec_in_app` (use this for data markers — volume/DB, robust to the serving layer); the lifecycle ops
|
||||
themselves come from `harness.generic` (`assert_serving`, `do_upgrade`, `do_backup`, `do_restore`).
|
||||
The harness forces `LETS_ENCRYPT_ENV=""` (no ACME), a unique short domain per run, and guarantees
|
||||
teardown.
|
||||
|
||||
## 3. Recipe-local tests (D4)
|
||||
|
||||
|
||||
127
docs/testing.md
Normal file
127
docs/testing.md
Normal file
@ -0,0 +1,127 @@
|
||||
# The cc-ci test architecture — generic suite + layered recipe overlays (Phase 1d)
|
||||
|
||||
Every recipe gets a **generic lifecycle test suite for free**. Recipe-specific tests *layer on top*
|
||||
of the generic default rather than being the only thing that runs. So `!testme` is meaningful on
|
||||
**any** recipe immediately (zero config), and adding recipe-specific coverage is a thin overlay.
|
||||
|
||||
## The model: tiers against one shared deployment
|
||||
|
||||
A run is a sequence of **tiers**. The orchestrator (`runner/run_recipe_ci.py`) deploys the app
|
||||
**once** and runs each tier against that single live deployment, then tears it down **once** in a
|
||||
`finally`. Lifecycle ops mutate the deployment **in place** — there is **no redeploy per tier**
|
||||
(asserted every run: `deploy-count = 1`).
|
||||
|
||||
```
|
||||
deploy ONCE (base version: the previous published version when an upgrade tier will run and one
|
||||
exists — so upgrade is a real previous→target; else the target / current PR head)
|
||||
→ INSTALL assertions (app already deployed: assert it really serves)
|
||||
→ UPGRADE abra app upgrade in place → target; assert reconverge + serving + the deployment MOVED
|
||||
→ BACKUP abra app backup create; assert a snapshot artifact (backup-capable recipes only)
|
||||
→ RESTORE abra app restore; assert healthy + serving (backup-capable recipes only)
|
||||
→ CUSTOM any non-lifecycle test_*.py (only if defined)
|
||||
teardown ONCE (in finally)
|
||||
```
|
||||
|
||||
Each tier is its own `pytest` invocation, so the run reports **per-operation** pass / fail / skip
|
||||
(`install / upgrade / backup / restore / custom`). The shared live domain is passed to each tier in
|
||||
`CCCI_APP_DOMAIN` and exposed by the `live_app` fixture; **tiers are assertion-only and never deploy
|
||||
or tear down** (that is the orchestrator's job).
|
||||
|
||||
## The generic default (recipe-agnostic)
|
||||
|
||||
Lives in the shared harness — `runner/harness/generic.py` + `tests/_generic/test_<op>.py` — so there
|
||||
is no per-recipe copy-paste:
|
||||
|
||||
- **install** (`generic.assert_serving`) — services converged (the app's *own* replicas are N/N) **and**
|
||||
a real HTTP(S) response in `HEALTH_OK` (which excludes 404, so a Traefik unmatched-router fallback
|
||||
fails) **and** the body isn't Traefik's default 404 page. A bounded poll (no bare `sleep`) so a
|
||||
state-mutating op settles, while a persistent failure still fails within the timeout. A CA-verified
|
||||
TLS handshake is also run as an **infra cert sanity check** (catches a lapsed/mis-rotated wildcard);
|
||||
it does **not** distinguish app-vs-fallback (Traefik serves the wildcard zone-wide) — that's the
|
||||
converged + non-404 check.
|
||||
- **upgrade** (`generic.do_upgrade`) — `abra app upgrade` in place to the target, then assert serving
|
||||
**and that the deployment actually moved** (the `coop-cloud.<stack>.version` label and/or image
|
||||
changed). The move-assertion makes a vacuous no-op upgrade impossible to pass.
|
||||
- **backup** (`generic.do_backup`) — `abra app backup create`; assert a snapshot artifact was produced
|
||||
(the `snapshot_id` in the create output). Honest limit: the generic verifies the *mechanism*, not
|
||||
app-specific data integrity (that's an overlay, below).
|
||||
- **restore** (`generic.do_restore`) — `abra app restore`; assert the app is healthy + serving after.
|
||||
|
||||
**Backup-capability** is auto-detected: a recipe is backup-capable iff a `compose*.yml` carries a
|
||||
truthy `backupbot.backup` label (override with `BACKUP_CAPABLE` in `recipe_meta.py`). For
|
||||
non-backup-capable recipes the backup/restore tiers are a clean **N/A skip** — not a failure.
|
||||
|
||||
## Recipe overlays — override or extend (the generic is always the default)
|
||||
|
||||
Convention: a recipe-specific tier is a file named exactly `test_install.py` / `test_upgrade.py` /
|
||||
`test_backup.py` / `test_restore.py`. **If present it OVERRIDES the generic for that op; if absent,
|
||||
the generic runs** (the invariant). Discovery looks in two locations, with this precedence:
|
||||
|
||||
```
|
||||
repo-local <recipe-repo>/tests/test_<op>.py (upstream-authoritative; wins same-name collisions)
|
||||
> cc-ci tests/<recipe>/test_<op>.py (CI-curated overlay)
|
||||
> generic tests/_generic/test_<op>.py (the floor; always present)
|
||||
```
|
||||
|
||||
- **Override** — a present `test_<op>.py` replaces the generic assertions for that op.
|
||||
- **Extend by composition** — an overlay may `from harness import generic` and call
|
||||
`generic.assert_serving(...)` / `generic.do_upgrade(...)` / `do_backup` / `do_restore`, then add its
|
||||
own recipe-specific assertions. (This is how every overlay reuses the generic op + serving check and
|
||||
layers data-continuity on top — no separate "extend" mechanism needed.)
|
||||
- **Custom (non-lifecycle) `test_*.py`** — any other `test_*.py` (e.g. `test_sso.py`) is **opt-in and
|
||||
additive**: it has no generic equivalent and runs only when present, discovered from **both**
|
||||
locations. Lifecycle names are excluded from the custom set.
|
||||
|
||||
Overlays are **assertion-only** and run against the shared deployment via the `live_app` fixture (so
|
||||
deploy-count stays 1). A data-continuity overlay reads/writes the app's *volume/DB* (via
|
||||
`lifecycle.exec_in_app`, robust to the serving layer), e.g.:
|
||||
|
||||
- `test_upgrade.py`: seed a marker → `generic.do_upgrade(...)` → assert the marker survived.
|
||||
- `test_backup.py`: seed "original" → `generic.do_backup(...)` → mutate to "mutated".
|
||||
- `test_restore.py`: `generic.do_restore(...)` → assert the marker is back to "original" (the backup
|
||||
tier's mutation persists on the shared deployment until the restore tier runs).
|
||||
|
||||
See `tests/custom-html/` (volume marker) and `tests/keycloak/`, `tests/matrix-synapse/`,
|
||||
`tests/lasuite-docs/` (admin-API / `db`-service markers) for worked examples.
|
||||
|
||||
## Custom install-steps hook (and the graceful-generic rule)
|
||||
|
||||
Some recipes need setup the generic flow won't do (pre-seed content, set an env/secret, run a one-off
|
||||
command). Provide a shell hook — `tests/<recipe>/install_steps.sh` (cc-ci) or repo-local
|
||||
`tests/install_steps.sh` (repo-local wins). The orchestrator runs it during the install tier **after
|
||||
`abra app new` + env defaults, before `abra app deploy`**, with env:
|
||||
|
||||
- `CCCI_APP_DOMAIN` — the run's app domain
|
||||
- `CCCI_RECIPE` — the recipe name
|
||||
- `CCCI_APP_ENV` — path to the app's `.env` (for `abra`-side edits)
|
||||
|
||||
**Graceful-generic rule:** a recipe with **no** hook still attempts the generic install. A recipe that
|
||||
genuinely needs a step will **fail the generic install — and that's the correct, reported outcome**
|
||||
(per-op `install: fail`); the fix is to add the step, not to special-case the harness. Worked example:
|
||||
`tests/custom-html-tiny/install_steps.sh` seeds an `index.html` into the static server's content
|
||||
volume — without it the generic install fails 404, with it it passes.
|
||||
|
||||
## How to add a recipe overlay (zero → some coverage)
|
||||
|
||||
1. The recipe is already testable with **zero config** — enrol it (poll list + mirror) and the generic
|
||||
suite runs (`docs/enroll-recipe.md`).
|
||||
2. To add recipe-specific coverage, drop a `tests/<recipe>/test_<op>.py` overlay (copy an existing one,
|
||||
e.g. `tests/keycloak/test_upgrade.py`). Reuse the generic op via `generic.do_<op>(...)` and add your
|
||||
assertions. Read/write app state through `lifecycle.exec_in_app` (volume/DB), not HTTP, for data
|
||||
checks. Set per-recipe knobs (health path, timeouts) in `recipe_meta.py`.
|
||||
3. If the recipe needs setup before it can serve, add `tests/<recipe>/install_steps.sh`.
|
||||
4. Never weaken or skip an assertion to make a run pass — a red tier is information.
|
||||
|
||||
Per-recipe config (`tests/<recipe>/recipe_meta.py`, all optional):
|
||||
|
||||
```python
|
||||
HEALTH_PATH = "/realms/master" # path that returns a healthy status (default "/")
|
||||
HEALTH_OK = (200,) # acceptable status codes (default 200/301/302)
|
||||
DEPLOY_TIMEOUT = 600 # seconds for services to converge (default 600)
|
||||
HTTP_TIMEOUT = 600 # seconds for the app to answer (default 300)
|
||||
BACKUP_CAPABLE = True # override backup-capability auto-detection (default: scan compose)
|
||||
EXTRA_ENV = {"KEY": "value"} # or EXTRA_ENV(domain) -> dict; extra .env keys set at deploy
|
||||
```
|
||||
|
||||
The harness self-tests for discovery/precedence live in `tests/unit/` (run: `cc-ci-run -m pytest
|
||||
tests/unit`); they are never picked up as overlays/custom tests.
|
||||
Reference in New Issue
Block a user