cc-ci/docs/testing.md

The cc-ci test architecture — generic suite + additive recipe overlays (Phase 1d + 1e)

Every recipe gets a generic lifecycle test suite for free — the floor under every run, always on by default. Recipe-specific tests layer additively on top: when a recipe ships an overlay for an op, the generic still runs alongside it (the floor is never silently lost). So !testme is meaningful on any recipe immediately (zero config), and adding recipe-specific coverage is a thin overlay that adds, it doesn't subtract.

Architectural invariant — generic-first, custom-additive (read this first)

This is the load-bearing principle of the whole test architecture. If you're maintaining cc-ci a year from now, this is the one rule that should still hold.

• Generic tests are simple and easily runnable. They are recipe-agnostic, depend only on the recipe being deployable (install / upgrade / backup / restore against the recipe alone), and ship as the floor for every recipe. No SSO provider, no external deps, no per-recipe state scaffolding — just "does this recipe deploy and lifecycle work?"
• Generic must not depend on custom. A custom test or a custom-tests setup (e.g. SSO/OIDC dep provisioning) can never be a precondition for the generic tier to pass. Concretely: the orchestrator runs all generic tiers (install → upgrade → backup → restore) against the recipe alone, with no deps deployed, then runs the setup_custom_tests step (deps + post-deps wiring) only after — and a failure there is isolated to the custom tier (tests tagged @pytest.mark.requires_deps skip with reason "deps-not-ready"; generic tier reports normally). See cc-ci-plan/plan-sso-dep-testing.md for the SSO-dep specifics.
• Custom tests are the thoroughness layer — and they cost more to maintain. They're more thorough (authenticated APIs, multi-app flows, version-specific browser selectors, helper scripts, state-management) and therefore take more maintenance: an SSO provider's admin API changes, a recipe's app-launch URL contract shifts between versions, a Socket.IO primitive needs to track upstream — these are real ongoing costs that the generic tier deliberately doesn't carry.
• A future maintainer can choose to focus on the generic tier alone and still get meaningful signal: every enrolled recipe gets some CI coverage from the generic floor, and the custom-additive layer can be scaled down or paused without breaking that floor. The choice of how much per-recipe depth to maintain is open to whoever owns cc-ci later — generic-only is a valid permanent operating mode.

If anything in this codebase ever asks you to make generic depend on custom (or to put a custom precondition before a generic tier), that's the signal it's drifted off the invariant — push back and restore the separation.

The model: tiers against one shared deployment

A run is a sequence of tiers. The orchestrator (runner/run_recipe_ci.py) deploys the app once and runs each tier against that single live deployment, then tears it down once in a finally. The orchestrator owns each mutating op (upgrade/backup/restore) and runs it exactly once; the assertion files (generic and overlay) evaluate the post-op state and never perform the op themselves. Asserted every run: deploy-count = 1 (one abra app new).

deploy ONCE  (base version: the previous published version when an upgrade tier will run and one
              exists — so upgrade is a real previous→PR-head; else the target / current PR head)
  → INSTALL    [optional pre_install seed]   then  generic + overlay assertions   (no op)
  → UPGRADE    [optional pre_upgrade seed]   then  abra app deploy --chaos to PR-head      (op once)
                                             then  generic + overlay assertions
  → BACKUP     [optional pre_backup seed]    then  abra app backup create                  (op once)
                                             then  generic + overlay assertions    (backup-capable only)
  → RESTORE    [optional pre_restore mutate] then  abra app restore                        (op once)
                                             then  generic + overlay assertions    (backup-capable only)
  → CUSTOM     any non-lifecycle test_*.py                                          (only if defined)
teardown ONCE (in finally)

Each assertion file is its own pytest invocation, so the run reports per-operation pass / fail / skip (install / upgrade / backup / restore / custom). The shared live domain is passed in CCCI_APP_DOMAIN and exposed by the live_app fixture; all assertion tiers are assertion-only and never deploy or tear down (that is the orchestrator's job). Op results an assertion needs (pre-upgrade identity, the produced backup snapshot_id) pass op→assertion via a run-scoped JSON state file at $CCCI_OP_STATE_FILE, read by generic.op_state().

The generic default (recipe-agnostic, the floor — Phase 1e HC3)

Lives in the shared harness — runner/harness/generic.py + tests/_generic/test_<op>.py — so there is no per-recipe copy-paste:

• install (generic.assert_serving) — services converged (the app's own replicas are N/N) and a real HTTP(S) response in HEALTH_OK (which excludes 404, so a Traefik unmatched-router fallback fails) and the body isn't Traefik's default 404 page. A bounded poll (no bare sleep) so a state-mutating op settles, while a persistent failure still fails within the timeout. A CA-verified TLS handshake also runs as an infra cert sanity check (catches a lapsed/mis-rotated wildcard); it does not distinguish app-vs-fallback (Traefik serves the wildcard zone-wide) — that's the converged + non-404 check.
• upgrade (generic.assert_upgraded) — assert serving after the orchestrator's chaos upgrade (HC1: abra app deploy --chaos of the PR-head checkout) and that the deployment is genuinely the code under test: when the intended PR-head commit is known, the deployed coop-cloud.<stack>.chaos-version label must match it — direct, non-vacuous proof. (A stale prev-checkout chaos redeploy would stamp prev's commit, not the PR-head, and fail here.) When head_ref is unknown, falls back to a move check (version/image/chaos changed vs pre-upgrade).
• backup (generic.assert_backup_artifact) — assert a snapshot artifact was produced (the snapshot_id captured by the orchestrator from abra app backup create). Honest limit: the generic verifies the mechanism, not app-specific data integrity (that's an overlay, below).
• restore (generic.assert_restore_healthy) — assert the app is healthy + serving after the orchestrator's restore op (assert_serving polls so the post-restore reconverge settles).

Backup-capability is auto-detected: a recipe is backup-capable iff a compose*.yml carries a truthy backupbot.backup label (override with BACKUP_CAPABLE in recipe_meta.py). For non-backup-capable recipes the backup/restore tiers are a clean N/A skip — not a failure.

Recipe overlays — additive (the generic floor is always on by default)

Convention: a recipe-specific tier is a file named exactly test_install.py / test_upgrade.py / test_backup.py / test_restore.py. When present it runs ALONGSIDE the generic for that op (both evaluate the shared post-op state); when absent, only the generic runs. Overlays are assertion-only — they never perform the op (the orchestrator owns it).

Overlay sources, in precedence order:

repo-local  <recipe-repo>/tests/test_<op>.py     (upstream-authoritative; gated by HC2 allowlist)
  >  cc-ci  tests/<recipe>/test_<op>.py           (CI-curated overlay)
  +  generic tests/_generic/test_<op>.py          (the floor; runs alongside by default)

Only ONE overlay source wins for a given op (repo-local > cc-ci); the generic floor runs in addition unless explicitly opted out.

Custom (non-lifecycle) test_*.py — any other test_*.py (e.g. test_sso.py) is opt-in and additive: it has no generic equivalent and runs only when present, discovered from both locations (repo-local gated by the HC2 allowlist).

Pre-op seed hooks (per-recipe ops.py)

A data-continuity overlay needs to seed state before the op (write a marker, create a DB row, etc.). Since the orchestrator owns the op, overlays place their seed in an optional per-recipe tests/<recipe>/ops.py:

# tests/<recipe>/ops.py
from harness import lifecycle

def pre_upgrade(domain, meta):
    # seed a marker before the harness performs the upgrade
    lifecycle.exec_in_app(domain, ["sh", "-c", "echo upgrade-survives > /path/marker"])

def pre_backup(domain, meta):
    # establish a known "original" state before the backup op captures it
    lifecycle.exec_in_app(domain, ["sh", "-c", "echo original > /path/marker"])

def pre_restore(domain, meta):
    # diverge from the backed-up state so a successful restore is observable
    lifecycle.exec_in_app(domain, ["sh", "-c", "echo mutated > /path/marker"])

The orchestrator imports ops.py in-process (with the recipe dir on sys.path, so it can import sibling helpers like kc_admin.py) and calls pre_<op>(domain, meta) immediately before performing the op. Then test_<op>.py asserts the post-op state. See tests/custom-html/ (volume marker), tests/keycloak/ (admin-API/realm), tests/matrix-synapse/, tests/lasuite-docs/ (psql in the db service) for worked examples.

Opting out of the generic floor

The generic runs additively by default. To skip it (e.g. when an overlay's recipe-specific check fully replaces the generic's mechanism check) set, in increasing specificity:

• env CCCI_SKIP_GENERIC=1 — skip generic for ALL ops (run-wide).
• env CCCI_SKIP_GENERIC_<OP>=1 — e.g. CCCI_SKIP_GENERIC_UPGRADE=1 — skip generic for that one op.
• declarative in recipe_meta.py — SKIP_GENERIC = ["upgrade"] (per-op) or SKIP_GENERIC = ["all"].

Opting out is per-recipe and visible in git — not a hidden global. Truthy = 1/true/yes/on.

Repo-local trust gate (HC2) — default-deny

PR-author-controlled code (a recipe repo's own tests/test_*.py, install_steps.sh, ops.py) runs on the CI host with /run/secrets/* present — an untrusted-code risk. By default the harness runs only cc-ci-authored overlays/hooks (tests/<recipe>/...) + the generic. Repo-local code is discovered-but-not-executed unless its recipe appears in tests/repo-local-approved.txt (a checked-in, git-auditable allowlist — one recipe name per line; # comments + blank lines ignored; a lone * is NOT a wildcard). To approve a recipe a cc-ci maintainer reviews its repo-local tests and adds the recipe name in a cc-ci PR (override the allowlist location with CCCI_REPO_LOCAL_APPROVED_FILE — used by tests + cold demonstrations).

The gate is centralized in runner/harness/discovery.py (repo_local_approved / _gated) so every discovery function (resolve_overlay_op, custom_tests, install_steps, pre_op_hook) honors it identically; unit tests (tests/unit/test_discovery.py) pin the behavior (approved-vs-not for every kind of code).

Custom install-steps hook (and the graceful-generic rule)

Some recipes need setup the generic flow won't do (pre-seed content, set an env/secret, run a one-off command). Provide a shell hook — tests/<recipe>/install_steps.sh (cc-ci) or repo-local tests/install_steps.sh (repo-local wins, gated by the HC2 allowlist). The orchestrator runs it during the install tier after abra app new + env defaults, before abra app deploy, with env:

• CCCI_APP_DOMAIN — the run's app domain
• CCCI_RECIPE — the recipe name
• CCCI_APP_ENV — path to the app's .env (for abra-side edits)

Graceful-generic rule: a recipe with no hook still attempts the generic install. A recipe that genuinely needs a step will fail the generic install — and that's the correct, reported outcome (per-op install: fail); the fix is to add the step, not to special-case the harness. Worked example: tests/custom-html-tiny/install_steps.sh seeds an index.html into the static server's content volume — without it the generic install fails 404, with it it passes.

The HC1 upgrade path — chaos to the PR-head code under test

Concretely, the upgrade tier:

1. base deployment is the previous published version (a clean pinned-tag deploy).
2. orchestrator captures head_ref (preferring $REF — the PR head sha; falls back to the recipe checkout HEAD for non-PR !testme).
3. on the upgrade tier: re-checkout the recipe to head_ref (the prev-tag base deploy reset the working tree), capture the pre-upgrade identity, then abra app deploy --chaos redeploys the running app at that checkout — in place, NOT a new install.
4. assert_upgraded (generic) asserts serving + that the deployed coop-cloud.<stack>.chaos-version matches head_ref — proving the PR-head code was deployed.

Reconciliation with the deploy-once guard: abra.deploy (chaos) is called directly, not through deploy_app, so _record_deploy() does not fire — deploy-count counts only abra app new installs and stays 1.

How to add a recipe overlay (zero → some coverage)

1. The recipe is already testable with zero config — enrol it (poll list + mirror) and the generic floor runs (docs/enroll-recipe.md).
2. To add recipe-specific coverage, drop tests/<recipe>/test_<op>.py (copy an existing one, e.g. tests/custom-html/test_upgrade.py). Assert the POST-op state — reading app state through lifecycle.exec_in_app (volume/DB) for data checks, not HTTP. Generic + your overlay both run.
3. If the overlay needs to seed PRE-op state (data-continuity markers, the backup→restore divergence), drop tests/<recipe>/ops.py with pre_upgrade/pre_backup/pre_restore(domain, meta).
4. If the recipe needs install-time setup, add tests/<recipe>/install_steps.sh.
5. Set per-recipe knobs (health path, timeouts, opt-out) in recipe_meta.py.
6. Never weaken or skip an assertion to make a run pass — a red tier is information.

Per-recipe config (tests/<recipe>/recipe_meta.py, all optional):

HEALTH_PATH = "/realms/master"   # path that returns a healthy status (default "/")
HEALTH_OK = (200,)               # acceptable status codes (default 200/301/302)
DEPLOY_TIMEOUT = 600             # seconds for services to converge (default 600)
HTTP_TIMEOUT = 600               # seconds for the app to answer (default 300)
BACKUP_CAPABLE = True            # override backup-capability auto-detection (default: scan compose)
EXTRA_ENV = {"KEY": "value"}     # or EXTRA_ENV(domain) -> dict; extra .env keys set at deploy
SKIP_GENERIC = ["upgrade"]       # per-recipe declarative opt-out from generic ops ("all" = every op)

The harness self-tests for discovery / precedence / the HC2 allowlist live in tests/unit/ (run: cc-ci-run -m pytest tests/unit); they are never picked up as overlays/custom tests.