status(1d): bootstrap Phase 1d — design recorded (tier model, override precedence, deploy-once), state files seeded

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
2026-05-27 23:06:21 +01:00
parent 6300cba503
commit a31095a087
4 changed files with 175 additions and 0 deletions

View File

@ -271,3 +271,64 @@ Plan §3's repo layout lists a `tests/_template/` "copy-to-add-a-recipe" dir. It
`recipe_meta.py` + the per-recipe test files."** This satisfies D5's "small, repeatable, documented
operation with no harness surgery" the same way (a concrete recipe is a better starting template than
an abstract skeleton that can drift). Recording per the Adversary's RL3 D5 advisory; not a blocker.
## Phase 1d — generic test suite + layered overlays (design, 2026-05-27)
SSOT: `cc-ci-plan/plan-phase1d-generic-test-suite.md`. Resolves the §6 open decisions.
- **Tier model & op/assertion split (the core call).** A run is a sequence of TIERS — install,
upgrade, backup, restore, custom — each = `generic default [overridden by a recipe overlay]`. The
**lifecycle OP** (deploy, upgrade, backup, restore) is owned by the **shared harness**
(`harness.generic` helpers), NOT duplicated in every test file. A tier's **test file** (generic or
overlay) carries the ASSERTIONS and calls the shared op helper. This keeps the op single-sourced
(DRY, DG7) and makes deploy-once trivial: only the orchestrator deploys/tears-down.
- **Override (not additive) — Builder's call (plan §6, operator leaned override).** For each
lifecycle op exactly ONE assertion file runs, by precedence:
**repo-local `tests/test_<op>.py` > cc-ci `tests/<recipe>/test_<op>.py` > generic
(`tests/_generic/test_<op>.py`)**. A present overlay REPLACES the generic for that op. **Invariant:
no overlay for an op ⇒ the generic runs** (so any recipe is testable with zero config). Repo-local
wins same-name collisions (upstream is authoritative, plan §2.5); cc-ci's overlay is the curated
fallback until upstream adopts it. **Extend-by-composition:** an overlay may
`from harness import generic` and call `generic.assert_serving(...)` / `generic.do_upgrade(...)`
then add its own assertions — so "extend" needs no separate mechanism.
- **Custom (non-lifecycle) `test_*.py`:** ALL discovered from BOTH locations run additively, opt-in
(no override, no generic equivalent) — e.g. `test_sso.py`.
- **Deploy ONCE, mutate in place (operator requirement, DG4.1).** The orchestrator deploys the app
ONCE, runs all tiers against that single live deployment (install asserts; upgrade does
`abra app upgrade` in place; backup/restore mutate in place; custom asserts), then ONE teardown in
`finally`. No per-tier/per-overlay `abra app new/deploy/undeploy`. A `CCCI_DEPLOY_COUNT` counter in
`lifecycle.deploy_app` is asserted == 1 per run (DG4.1 evidence).
- **Deployment-sharing scope & base version (§6 open).** One deployment for the whole lifecycle.
Base version deployed once = the **previous published version** when an upgrade tier will run and a
previous exists (so upgrade goes previous→target in place), **else the target** (current/$REF).
Recipe with only one published version ⇒ upgrade tier is a clean **SKIP** (nothing to upgrade
from). Standalone generic-install demo (no PR) deploys current.
- **Fail handling across shared tiers (§6 open):** install failing (app never serves) **fail-fasts**
the run (later tiers can't meaningfully run on a dead deployment) and they report **error/skip**;
upgrade/backup/restore failures are recorded per-op but do not abort the remaining independent
tiers where they can still run. Teardown always runs.
- **Backup-capability detection (DG3, §6 open):** auto — scan the recipe's `compose*.yml` for a
`backupbot.backup` label (verified present in custom-html). `recipe_meta.BACKUP_CAPABLE` (bool)
overrides the auto-detect. Not capable ⇒ backup+restore tiers are **N/A (skip)**, not failures.
- **Custom install-steps hook (DG5, §6 open):** a shell hook — `tests/<recipe>/install_steps.sh`
(cc-ci) or repo-local `tests/install_steps.sh` — run by the orchestrator during the install tier
AFTER `abra app new` + env defaults but BEFORE `abra app deploy`, with env `CCCI_APP_DOMAIN`,
`CCCI_RECIPE`, `CCCI_APP_ENV` (path to the app .env). Chosen over a fixture/declarative field as the
simplest thing the harness runs uniformly (can `abra app secret insert`, set env, seed). Graceful
rule: a recipe with NO hook still attempts the generic install; if it genuinely needs a step it
FAILS the generic install (reported per-op) — that is correct, not a harness bug.
- **Per-op result vocabulary (Phase-3 feed):** `pass | fail | skip(N/A) | error`. The orchestrator
prints a per-op summary line per run (feeds DG6 + Phase-3 level).
- **Discovery layout:** cc-ci overlays/custom/hook live in `tests/<recipe>/`; repo-local in the
recipe repo's `tests/` (snapshotted after fetch, per the existing volatile-checkout handling).
Generic tier files live in `tests/_generic/` (assertion-only, use the shared live-deployment
fixtures).