status(1d): bootstrap Phase 1d — design recorded (tier model, override precedence, deploy-once), state files seeded
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
@ -271,3 +271,64 @@ Plan §3's repo layout lists a `tests/_template/` "copy-to-add-a-recipe" dir. It
|
||||
`recipe_meta.py` + the per-recipe test files."** This satisfies D5's "small, repeatable, documented
|
||||
operation with no harness surgery" the same way (a concrete recipe is a better starting template than
|
||||
an abstract skeleton that can drift). Recording per the Adversary's RL3 D5 advisory; not a blocker.
|
||||
|
||||
## Phase 1d — generic test suite + layered overlays (design, 2026-05-27)
|
||||
|
||||
SSOT: `cc-ci-plan/plan-phase1d-generic-test-suite.md`. Resolves the §6 open decisions.
|
||||
|
||||
- **Tier model & op/assertion split (the core call).** A run is a sequence of TIERS — install,
|
||||
upgrade, backup, restore, custom — each = `generic default [overridden by a recipe overlay]`. The
|
||||
**lifecycle OP** (deploy, upgrade, backup, restore) is owned by the **shared harness**
|
||||
(`harness.generic` helpers), NOT duplicated in every test file. A tier's **test file** (generic or
|
||||
overlay) carries the ASSERTIONS and calls the shared op helper. This keeps the op single-sourced
|
||||
(DRY, DG7) and makes deploy-once trivial: only the orchestrator deploys/tears-down.
|
||||
|
||||
- **Override (not additive) — Builder's call (plan §6, operator leaned override).** For each
|
||||
lifecycle op exactly ONE assertion file runs, by precedence:
|
||||
**repo-local `tests/test_<op>.py` > cc-ci `tests/<recipe>/test_<op>.py` > generic
|
||||
(`tests/_generic/test_<op>.py`)**. A present overlay REPLACES the generic for that op. **Invariant:
|
||||
no overlay for an op ⇒ the generic runs** (so any recipe is testable with zero config). Repo-local
|
||||
wins same-name collisions (upstream is authoritative, plan §2.5); cc-ci's overlay is the curated
|
||||
fallback until upstream adopts it. **Extend-by-composition:** an overlay may
|
||||
`from harness import generic` and call `generic.assert_serving(...)` / `generic.do_upgrade(...)`
|
||||
then add its own assertions — so "extend" needs no separate mechanism.
|
||||
|
||||
- **Custom (non-lifecycle) `test_*.py`:** ALL discovered from BOTH locations run additively, opt-in
|
||||
(no override, no generic equivalent) — e.g. `test_sso.py`.
|
||||
|
||||
- **Deploy ONCE, mutate in place (operator requirement, DG4.1).** The orchestrator deploys the app
|
||||
ONCE, runs all tiers against that single live deployment (install asserts; upgrade does
|
||||
`abra app upgrade` in place; backup/restore mutate in place; custom asserts), then ONE teardown in
|
||||
`finally`. No per-tier/per-overlay `abra app new/deploy/undeploy`. A `CCCI_DEPLOY_COUNT` counter in
|
||||
`lifecycle.deploy_app` is asserted == 1 per run (DG4.1 evidence).
|
||||
|
||||
- **Deployment-sharing scope & base version (§6 open).** One deployment for the whole lifecycle.
|
||||
Base version deployed once = the **previous published version** when an upgrade tier will run and a
|
||||
previous exists (so upgrade goes previous→target in place), **else the target** (current/$REF).
|
||||
Recipe with only one published version ⇒ upgrade tier is a clean **SKIP** (nothing to upgrade
|
||||
from). Standalone generic-install demo (no PR) deploys current.
|
||||
|
||||
- **Fail handling across shared tiers (§6 open):** install failing (app never serves) **fail-fasts**
|
||||
the run (later tiers can't meaningfully run on a dead deployment) and they report **error/skip**;
|
||||
upgrade/backup/restore failures are recorded per-op but do not abort the remaining independent
|
||||
tiers where they can still run. Teardown always runs.
|
||||
|
||||
- **Backup-capability detection (DG3, §6 open):** auto — scan the recipe's `compose*.yml` for a
|
||||
`backupbot.backup` label (verified present in custom-html). `recipe_meta.BACKUP_CAPABLE` (bool)
|
||||
overrides the auto-detect. Not capable ⇒ backup+restore tiers are **N/A (skip)**, not failures.
|
||||
|
||||
- **Custom install-steps hook (DG5, §6 open):** a shell hook — `tests/<recipe>/install_steps.sh`
|
||||
(cc-ci) or repo-local `tests/install_steps.sh` — run by the orchestrator during the install tier
|
||||
AFTER `abra app new` + env defaults but BEFORE `abra app deploy`, with env `CCCI_APP_DOMAIN`,
|
||||
`CCCI_RECIPE`, `CCCI_APP_ENV` (path to the app .env). Chosen over a fixture/declarative field as the
|
||||
simplest thing the harness runs uniformly (can `abra app secret insert`, set env, seed). Graceful
|
||||
rule: a recipe with NO hook still attempts the generic install; if it genuinely needs a step it
|
||||
FAILS the generic install (reported per-op) — that is correct, not a harness bug.
|
||||
|
||||
- **Per-op result vocabulary (Phase-3 feed):** `pass | fail | skip(N/A) | error`. The orchestrator
|
||||
prints a per-op summary line per run (feeds DG6 + Phase-3 level).
|
||||
|
||||
- **Discovery layout:** cc-ci overlays/custom/hook live in `tests/<recipe>/`; repo-local in the
|
||||
recipe repo's `tests/` (snapshotted after fetch, per the existing volatile-checkout handling).
|
||||
Generic tier files live in `tests/_generic/` (assertion-only, use the shared live-deployment
|
||||
fixtures).
|
||||
|
||||
Reference in New Issue
Block a user