diff --git a/machine-docs/BACKLOG-1d.md b/machine-docs/BACKLOG-1d.md new file mode 100644 index 0000000..bbe82ed --- /dev/null +++ b/machine-docs/BACKLOG-1d.md @@ -0,0 +1,39 @@ +# BACKLOG — Phase 1d + +## Build backlog (Builder-only) + +### G0 — Generic install + deploy-once orchestrator (DG1) +- [ ] `runner/harness/generic.py`: generic assertion helpers (`assert_serving` — real HTTP, not + Traefik fallback/default cert) + op helpers (`do_upgrade`, `do_backup`, `do_restore`) + + `backup_capable(recipe)` (scan compose for backupbot.backup). +- [ ] `runner/harness/discovery.py`: per-op overlay resolution (repo-local > cc-ci > generic), + custom-test discovery (both locations, additive), install-steps hook discovery. +- [ ] `tests/_generic/`: assertion-only generic tier files (test_install/upgrade/backup/restore.py). +- [ ] Refactor `run_recipe_ci.py` → deploy-once: deploy base version once, run tiers in order against + the shared deployment, one teardown in finally; per-op result summary. +- [ ] Refactor `tests/conftest.py` fixtures to expose the shared live deployment (no per-tier deploy). +- [ ] Deploy-count guard (`CCCI_DEPLOY_COUNT`) in `lifecycle.deploy_app`; assert ==1 per run. +- [ ] Prove generic install green on custom-html-tiny (no cc-ci/repo-local tests). → claim G0. + +### G1 — Generic upgrade + backup/restore (DG2, DG3) +- [ ] Generic upgrade tier: previous→target in place; reconverge + serving. +- [ ] Generic backup/restore tiers gated on backup-capability; clean N/A skip otherwise. +- [ ] Prove on a backup-capable recipe (custom-html: has backupbot labels). + +### G2 — Layering + discovery + precedence (DG4, DG4.1) +- [ ] Migrate an existing recipe's tests to the new assertion-only overlay contract as the proof. +- [ ] Prove override (overlay replaces generic) + extend-by-composition; no redeploy (deploy-count==1). + +### G3 — Custom install-steps hook + graceful-generic (DG5) +- [ ] install_steps.sh hook run during install tier (after app new+env, before deploy). +- [ ] Proof: a recipe needing a step FAILS generic install without it; PASSES with it. + +### G4 — !testme e2e + per-op reporting + docs + cold verify (DG6, DG7, DG8) +- [ ] !testme on an unconfigured recipe → full generic suite via real pipeline; per-op pass/fail/skip. +- [ ] Migrate remaining recipe tests to the new contract so nothing regresses (DG7). +- [ ] docs/: generic suite, overlay convention (names/locations/precedence), install-steps hook, + how to add an overlay. +- [ ] Request Adversary cold-verify DG1–DG8 → flip STATUS-1d to ## DONE. + +## Adversary findings (Adversary-only) +(none yet) diff --git a/machine-docs/DECISIONS.md b/machine-docs/DECISIONS.md index d320264..1194d45 100644 --- a/machine-docs/DECISIONS.md +++ b/machine-docs/DECISIONS.md @@ -271,3 +271,64 @@ Plan §3's repo layout lists a `tests/_template/` "copy-to-add-a-recipe" dir. It `recipe_meta.py` + the per-recipe test files."** This satisfies D5's "small, repeatable, documented operation with no harness surgery" the same way (a concrete recipe is a better starting template than an abstract skeleton that can drift). Recording per the Adversary's RL3 D5 advisory; not a blocker. + +## Phase 1d — generic test suite + layered overlays (design, 2026-05-27) + +SSOT: `cc-ci-plan/plan-phase1d-generic-test-suite.md`. Resolves the §6 open decisions. + +- **Tier model & op/assertion split (the core call).** A run is a sequence of TIERS — install, + upgrade, backup, restore, custom — each = `generic default [overridden by a recipe overlay]`. The + **lifecycle OP** (deploy, upgrade, backup, restore) is owned by the **shared harness** + (`harness.generic` helpers), NOT duplicated in every test file. A tier's **test file** (generic or + overlay) carries the ASSERTIONS and calls the shared op helper. This keeps the op single-sourced + (DRY, DG7) and makes deploy-once trivial: only the orchestrator deploys/tears-down. + +- **Override (not additive) — Builder's call (plan §6, operator leaned override).** For each + lifecycle op exactly ONE assertion file runs, by precedence: + **repo-local `tests/test_.py` > cc-ci `tests//test_.py` > generic + (`tests/_generic/test_.py`)**. A present overlay REPLACES the generic for that op. **Invariant: + no overlay for an op ⇒ the generic runs** (so any recipe is testable with zero config). Repo-local + wins same-name collisions (upstream is authoritative, plan §2.5); cc-ci's overlay is the curated + fallback until upstream adopts it. **Extend-by-composition:** an overlay may + `from harness import generic` and call `generic.assert_serving(...)` / `generic.do_upgrade(...)` + then add its own assertions — so "extend" needs no separate mechanism. + +- **Custom (non-lifecycle) `test_*.py`:** ALL discovered from BOTH locations run additively, opt-in + (no override, no generic equivalent) — e.g. `test_sso.py`. + +- **Deploy ONCE, mutate in place (operator requirement, DG4.1).** The orchestrator deploys the app + ONCE, runs all tiers against that single live deployment (install asserts; upgrade does + `abra app upgrade` in place; backup/restore mutate in place; custom asserts), then ONE teardown in + `finally`. No per-tier/per-overlay `abra app new/deploy/undeploy`. A `CCCI_DEPLOY_COUNT` counter in + `lifecycle.deploy_app` is asserted == 1 per run (DG4.1 evidence). + +- **Deployment-sharing scope & base version (§6 open).** One deployment for the whole lifecycle. + Base version deployed once = the **previous published version** when an upgrade tier will run and a + previous exists (so upgrade goes previous→target in place), **else the target** (current/$REF). + Recipe with only one published version ⇒ upgrade tier is a clean **SKIP** (nothing to upgrade + from). Standalone generic-install demo (no PR) deploys current. + +- **Fail handling across shared tiers (§6 open):** install failing (app never serves) **fail-fasts** + the run (later tiers can't meaningfully run on a dead deployment) and they report **error/skip**; + upgrade/backup/restore failures are recorded per-op but do not abort the remaining independent + tiers where they can still run. Teardown always runs. + +- **Backup-capability detection (DG3, §6 open):** auto — scan the recipe's `compose*.yml` for a + `backupbot.backup` label (verified present in custom-html). `recipe_meta.BACKUP_CAPABLE` (bool) + overrides the auto-detect. Not capable ⇒ backup+restore tiers are **N/A (skip)**, not failures. + +- **Custom install-steps hook (DG5, §6 open):** a shell hook — `tests//install_steps.sh` + (cc-ci) or repo-local `tests/install_steps.sh` — run by the orchestrator during the install tier + AFTER `abra app new` + env defaults but BEFORE `abra app deploy`, with env `CCCI_APP_DOMAIN`, + `CCCI_RECIPE`, `CCCI_APP_ENV` (path to the app .env). Chosen over a fixture/declarative field as the + simplest thing the harness runs uniformly (can `abra app secret insert`, set env, seed). Graceful + rule: a recipe with NO hook still attempts the generic install; if it genuinely needs a step it + FAILS the generic install (reported per-op) — that is correct, not a harness bug. + +- **Per-op result vocabulary (Phase-3 feed):** `pass | fail | skip(N/A) | error`. The orchestrator + prints a per-op summary line per run (feeds DG6 + Phase-3 level). + +- **Discovery layout:** cc-ci overlays/custom/hook live in `tests//`; repo-local in the + recipe repo's `tests/` (snapshotted after fetch, per the existing volatile-checkout handling). + Generic tier files live in `tests/_generic/` (assertion-only, use the shared live-deployment + fixtures). diff --git a/machine-docs/JOURNAL-1d.md b/machine-docs/JOURNAL-1d.md new file mode 100644 index 0000000..73f15cb --- /dev/null +++ b/machine-docs/JOURNAL-1d.md @@ -0,0 +1,29 @@ +# JOURNAL — Phase 1d (append-only) + +## 2026-05-27 — Bootstrap Phase 1d + +Read SSOT `plan-phase1d-generic-test-suite.md` + plan.md §6.1/§7/§9. Studied the post-1b codebase: +`runner/run_recipe_ci.py` (per-stage pytest, currently deploy-per-stage), `tests/conftest.py` +(fixtures `deployed_app`/`deployed`/`old_app` each deploy+teardown), `runner/harness/{lifecycle,abra,naming}.py`, +and existing recipe tests (custom-html/keycloak/etc.). + +Access re-verified (bootstrap, new phase): +``` +$ ssh cc-ci 'hostname && whoami && nixos-version' +nixos / root / 24.11.20250630.50ab793 (Vicuna) +$ ssh cc-ci 'abra --version' -> abra version 0.13.0-beta-06a57de +$ ssh cc-ci 'docker stack ls' -> traefik, drone, ccci-bridge, ccci-dashboard, backups all up +$ ssh cc-ci 'grep -ri backupbot ~/.abra/recipes/custom-html/' + compose.yml: backupbot.backup=true ; backupbot.backup.path=/usr/share/nginx/html +$ curl -u bot ... /repos/recipe-maintainers/custom-html-tiny -> 200 (mirrored) +``` +So: backup-capability is detectable by scanning compose for `backupbot.backup`; custom-html-tiny is +mirrored and has NO cc-ci tests dir → it's the DG1 pure-generic target. + +**Design recorded in DECISIONS.md (Phase 1d section).** Key calls: tier model with the lifecycle OP +owned by the shared harness (test files = assertions only); OVERRIDE precedence repo-local > cc-ci > +generic with extend-by-composition; deploy-ONCE with a deploy-count guard; base version = previous +(when upgrade runs) else target; backup-capability auto-detect; install-steps shell hook. + +Seeded STATUS-1d / BACKLOG-1d / JOURNAL-1d. Next: implement G0 (generic.py + discovery.py + +tests/_generic/ + deploy-once orchestrator), then verify generic install green on custom-html-tiny. diff --git a/machine-docs/STATUS-1d.md b/machine-docs/STATUS-1d.md new file mode 100644 index 0000000..0206dd0 --- /dev/null +++ b/machine-docs/STATUS-1d.md @@ -0,0 +1,46 @@ +# STATUS — Phase 1d (generic test suite + layered recipe overlays) + +**Phase plan (SSOT):** `/srv/cc-ci/cc-ci-plan/plan-phase1d-generic-test-suite.md` +**Loop state for THIS phase:** STATUS-1d / BACKLOG-1d / REVIEW-1d / JOURNAL-1d (DECISIONS.md shared). +The repo's STATUS.md/BACKLOG.md/REVIEW.md (Phase 1) and STATUS-1b/1c (DONE) are HISTORY, not this +phase's state. + +## Phase +Phase 1d runs after Phase 1b (DONE) and before Phase 2. It is the **test-architecture foundation**: +every recipe gets a generic lifecycle suite for free; recipe-specific tests layer on top +(override-or-extend). Bounded — build the architecture + prove it on a couple of recipes; full +per-recipe overlay authoring is Phase 2. + +## Definition of Done (Phase 1d) — DG1–DG8, each Adversary cold-verified in REVIEW-1d +- [ ] **DG1** — Generic INSTALL test (recipe-agnostic): app new→deploy→converged→really serving + (real HTTP(S), not Traefik fallback). Green on a simple recipe with no cc-ci/repo-local tests. +- [ ] **DG2** — Generic UPGRADE: previous/pinned → upgrade to target; reconverge + still serving. +- [ ] **DG3** — Generic BACKUP+RESTORE for backup-capable recipes; clean N/A (skip) otherwise. +- [ ] **DG4** — Layering (override-or-extend; generic is the default); discovery + cc-ci/repo-local + precedence settled in DECISIONS. Invariant: no overlay for an op ⇒ generic runs. +- [ ] **DG4.1** — Overlays reuse the deployment: ONE deploy + ONE teardown per run; no extra + new/deploy/undeploy (assert via deploy-count). +- [ ] **DG5** — Custom install-steps hook + graceful-generic rule; fail-without / pass-with proof. +- [ ] **DG6** — `!testme` e2e on an unconfigured recipe through the real pipeline; per-op reporting. +- [ ] **DG7** — Real, DRY, clean: no softened/skip/xfail assertions; generic in the shared harness; + teardown always; respects MAX_TESTS. +- [ ] **DG8** — Documented (docs/ explains the generic suite, overlay convention, hook) + cold-verify. + +## Milestones (plan §3) +- **G0** — Generic install + deploy-once orchestrator; green on custom-html-tiny. *Accept: DG1.* +- **G1** — Generic upgrade + backup/restore. *Accept: DG2, DG3.* +- **G2** — Layering + discovery + precedence. *Accept: DG4, DG4.1.* +- **G3** — Custom install-steps hook + graceful-generic. *Accept: DG5.* +- **G4** — `!testme` e2e + per-op reporting + docs + cold verify. *Accept: DG6, DG7, DG8 → DONE.* + +## In flight +**G0 — generic install + deploy-once orchestrator.** Design recorded in DECISIONS.md (tier model, +override precedence, deploy-once, backup-capability auto-detect, install-steps shell hook). Building +`harness/generic.py` + `harness/discovery.py` + new deploy-once `run_recipe_ci.py` + `tests/_generic/`. + +## Gate +(none yet — will claim G0 when generic install is green on custom-html-tiny) + +## Blocked +(none) — bootstrap access re-verified @2026-05-27: ssh cc-ci ok (root, NixOS 24.11), abra 0.13.0-beta, +5 infra stacks up (traefik/drone/bridge/dashboard/backups), custom-html-tiny mirrored.