diff --git a/machine-docs/BACKLOG-2.md b/machine-docs/BACKLOG-2.md new file mode 100644 index 0000000..3a019ee --- /dev/null +++ b/machine-docs/BACKLOG-2.md @@ -0,0 +1,88 @@ +# BACKLOG — Phase 2 (per-recipe test authoring) + +Phase-namespaced backlog. Builder edits `## Build backlog`; Adversary edits `## Adversary findings`. +Phase plan: `/srv/cc-ci/cc-ci-plan/plan-phase2-recipe-tests.md` + +## Build backlog + +### Q0 — Harness additions +- [ ] **Q0.1** — Vendor `recipe-maintainer/utils/tests/helpers.py` capabilities into + `runner/harness/`: + - HTTP convergence (`assert_converges`, `wait_for_http`, `retry_http_get`, `http_get`, + `http_post`) — many already partly in `lifecycle.py`/`generic.py`; add the missing pieces + + consolidate (no duplicate logic). + - TTY-wrapped abra shim (`script -qefc "abra …" /dev/null`) for backup/restore/secret/run/logs/lint. + - Backup data-integrity primitive (volume/DB seed → marker survival check). +- [ ] **Q0.2** — Discovery: recurse into `tests//functional/` and `tests//playwright/` + so per-plan `§4.1` directory layout is honored. Update `discovery.custom_tests` + `enroll-recipe.md`. +- [ ] **Q0.3** — PARITY.md template + a worked custom-html example + (`tests/custom-html/PARITY.md`). Port custom-html health_check.py to a functional test. +- [ ] **Q0.4** — Dependency resolver harness primitive (read `tests//recipe.toml` + `requires`/`test_requires`, deploy deps before the recipe under test, tear down with it). Mind + `MAX_TESTS`/node budget; sequence heavy ones. Defer SSO-realm-setup wiring to Q2. +- [ ] **Q0.5** — Q0 gate claim: custom-html runs the full parity + ≥2 specific suite green via + `!testme`; PARITY.md complete for custom-html; harness changes recorded in DECISIONS.md. + +### Q1 — Pattern proof (custom-html + n8n) +- [ ] **Q1.1** — custom-html: ≥2 NEW recipe-specific functional tests (beyond parity + `health_check`). Candidates from plan §4.3: "serve/persist content: write content, fetch it back" + — custom-html serves the `/usr/share/nginx/html` volume, so a content round-trip + a content-type + header check are appropriate. Playwright already exists in `tests/custom-html/test_install.py`. +- [ ] **Q1.2** — n8n: enroll under cc-ci. Port `recipe-info/n8n/tests/health_check.py` → + `tests/n8n/functional/health_check.py`. Add ≥2 specific tests: (a) create a workflow via API, + execute it, assert result; (b) the workflow persists across an upgrade. PARITY.md filled. +- [ ] **Q1.3** — Real backup data-integrity for n8n: seed a workflow → backup → wipe → restore → list + workflows, prove the seeded one survived. (custom-html already has this pattern from 1e.) +- [ ] **Q1.4** — Q1 gate: both recipes green via `!testme`; both PARITY.md complete. + +### Q2 — SSO providers (keycloak + authentik) +- [ ] **Q2.1** — keycloak: port `tests/keycloak/oidc_integration.py` (the dependent-recipe test) and + `tests/health_check.py`. Add specific tests from plan §4.3 (realm+client via admin API; password + and client-credentials token grants; JWT claims). +- [ ] **Q2.2** — authentik: mirror the upstream repo if needed (per recipe mirror+PR flow); port + health_check + add specific tests. +- [ ] **Q2.3** — Reusable SSO-setup/OIDC-flow harness primitive: deploy provider → setup realm/client/ + test-user (port `recipe-info//setup__integration.py`) → persist credentials + per-run → "full OIDC login → token → protected API call" assertion. Implement once in + `runner/harness/`; reused by every SSO-dependent recipe. +- [ ] **Q2.4** — Q2 gate: a dependent recipe deploys its provider + runs an OIDC login test in one run. + +### Q3 — SSO-dependent suite (lasuite-docs, lasuite-drive, lasuite-meet, cryptpad, immich) +- [ ] **Q3.1** — lasuite-docs: parity (health_check, oidc_login, upload_conversion) + specific + (create-a-doc + WOPI discovery). +- [ ] **Q3.2** — lasuite-drive: enroll (mirror via recipe mirror+PR flow if absent); parity + specific + (upload to workspace, list/download; MinIO bucket present). +- [ ] **Q3.3** — lasuite-meet: parity (health_check, oidc_login, meeting_flow, webrtc-media, + webrtc-relay) + specific (create-a-room, two-user LiveKit token issuance, ICE-candidate gathering). +- [ ] **Q3.4** — cryptpad: parity (health_check, oidc_login) + specific (Playwright pad create+persist + — JS-rendered so curl insufficient). +- [ ] **Q3.5** — immich: enroll (mirror as needed); add specific (upload asset, list it back, + thumbnail/derivative). +- [ ] **Q3.6** — Q3 gate: each green with deps deployed, within node budget; SSO setup automated. + +### Q4 — Remaining recipes +- [ ] **Q4.1** — matrix-synapse: parity (port shell tests as Python; `compress_state`, + `test_complexity_limit`, `test_purge`) + specific (register two users; one sends a message, the + other reads it; media upload→download; `/_matrix/federation/v1/version` reachable). +- [ ] **Q4.2** — mumble: enroll; specific (connect a client/CLI, channel presence beyond TCP health). +- [ ] **Q4.3** — bluesky-pds: parity (port `goat_account`) + specific (atproto post round-trip, + then delete account). +- [ ] **Q4.4** — ghost: enroll; specific (create-a-post round-trip). +- [ ] **Q4.5** — mattermost-lts: enroll; specific (create-a-message round-trip). +- [ ] **Q4.6** — discourse: enroll; specific (create-a-topic round-trip). +- [ ] **Q4.7** — plausible: enroll; specific (track a test event, query it back). +- [ ] **Q4.8** — uptime-kuma: enroll; specific (create a monitor, list it). +- [ ] **Q4.9** — mailu: enroll; specific (create a mailbox, send/receive verification). +- [ ] **Q4.10** — drone: enroll; specific (create/list builds via API). +- [ ] **Q4.11** — Q4 gate: each recipe green with parity + specific. + +### Q5 — Completeness + docs +- [ ] **Q5.1** — `docs/enroll-recipe.md` updated with the per-recipe test contract (§4.1), the + `functional/` and `playwright/` subdirectory layout, the PARITY.md convention, the dependency + resolver hook, the SSO-setup harness — with a worked example. +- [ ] **Q5.2** — Adversary samples a subset and cold-verifies parity tables + specific tests are real + (not health-only, not skipped). NO weakened test, no corners cut (P7). +- [ ] **Q5.3** — Phase 2 `## DONE` after all P1–P8 Adversary cold-verified PASS, no standing VETO. + +## Adversary findings +(empty — Adversary writes here) diff --git a/machine-docs/DECISIONS.md b/machine-docs/DECISIONS.md index cf5d201..5a41452 100644 --- a/machine-docs/DECISIONS.md +++ b/machine-docs/DECISIONS.md @@ -379,3 +379,64 @@ Three operator-review corrections to the Phase-1d shared harness, settled here ( `coop-cloud..chaos`/`.chaos-version`) — the chaos label IS the proof PR-head was deployed. Non-PR `!testme` (no SRC/REF): "PR head" = the catalogue current checkout, so upgrade is prev→current — still a genuine move via chaos. (Exact chaos label name verified on the live abra during E2.) + +## Phase 2 — per-recipe test authoring (design, 2026-05-28) +Inherits the Phase 1d/1e shared-deployment + additive-overlay + op/assertion-split model. Phase 2 +adds **content**, not infra, with a few small harness primitives ported from +`references/recipe-maintainer/utils/tests/helpers.py`. + +- **Per-recipe layout (per plan §4.1).** The cc-ci `tests//` dir continues to use the + Phase-1d/1e overlays at the top level (`test_install.py`, `test_upgrade.py`, `test_backup.py`, + `test_restore.py`, `ops.py`, `recipe_meta.py`, optional `install_steps.sh`). NEW Phase-2 + subdirectories: + - `tests//functional/` — parity-port tests (one per recipe-maintainer `tests/*.py`) + + ≥2 NEW recipe-specific functional tests (P2/P3). Each file is `test_*.py` (pytest-discoverable); + each parity port carries a **`SOURCE = "recipe-info//tests/"`** comment near the + top so the audit trail is in the file, not just in PARITY.md. + - `tests//playwright/` — browser flows (P6) where the app's UX is a UI flow. Same + `test_*.py` convention; each file imports `playwright.sync_api`. + - `tests//PARITY.md` — required mapping table (P2) with one row per + recipe-info parity test: `| recipe-maintainer file | cc-ci file | what's verified | status |`. + A deliberate non-port is a documented row in DECISIONS.md (linked from PARITY.md), not a silent + omission. +- **Discovery for the new subdirs.** `runner/harness/discovery.custom_tests` recurses into + `tests//functional/` and `tests//playwright/` (in addition to the top-level glob), + so Phase-2 functional tests run as part of the **custom** stage automatically. Repo-local (HC2) + gate still applies if the recipe is approved; otherwise only cc-ci's own functional/ + playwright/ + run. The top-level `test_install.py`/etc. continue to drive the lifecycle overlays — the + `functional/` + `playwright/` files are **always custom-stage**, never lifecycle (so they don't + perform an op; they assert against the post-install live deployment). +- **Vendored helpers in `runner/harness/`.** Capabilities ported from `recipe-maintainer/utils/tests/ + helpers.py` (cc-ci is self-contained at runtime — does NOT import recipe-maintainer's workspace, + per plan §8 default): + - `harness.http` — `http_get(url, headers=, timeout=) -> (status, json_or_None)`, + `http_post(...)`, `retry_http_get(url, timeout=, **)`, `wait_for_http(url, label, max_wait=)`, + `assert_converges(fn, description, max_wait=, interval=)`. (Several variants exist + `lifecycle.http_fetch/http_get/http_body` already; the harness.http module is the **canonical** + Phase-2 HTTP API for tests; lifecycle.* helpers stay for infra-level checks.) + - `harness.abra_tty` — `script -qefc "abra …" /dev/null` wrapper for the abra commands that + require a TTY (backup/restore/secret/run/logs/lint), used by parity tests that drive abra + directly. Lifecycle already exposes typed wrappers — this is for tests that need raw shell-abra. + - `harness.deps` — dependency resolver primitive. Reads `tests//recipe.toml` + (`requires` / `test_requires`), deploys each declared dep via the same `lifecycle.deploy_app` + + `wait_healthy` path (so the dep is a real `-<6hex>.ci.commoninternet.net` on the + same swarm), persists per-run, tears down with the parent in the orchestrator's `finally`. + Heavy recipes sequence sequentially; `MAX_TESTS`/node budget is the cap. + - `harness.sso` — OIDC-flow primitive (Q2 deliverable). Given a deployed provider domain and a + recipe-defined realm/client/test-user, performs the full "deploy provider → setup realm/client + via admin API → obtain access token (password + client-credentials grants) → assert protected + API call accepts it" assertion. Reusable by every SSO-dependent recipe (cryptpad, lasuite-*, + immich, etc.). Setup scripts ported from `recipe-info//setup__integration.py`. + - `harness.data_integrity` — backup data-integrity primitive: a recipe-aware "seed a marker + → backup → mutate → restore → assert seeded marker survived" helper around `lifecycle.exec_in_app` + / `http_get` (the recipe chooses the marker mechanism, the helper guarantees the pattern). +- **Run-scoped credentials for SSO/recipe-specific tests** (plan §4.4 class-B). Generated secrets + (realm/client/test-user passwords, API tokens) persist for the run via the existing + `runs//` mechanism (Phase 1d). Destroyed at teardown alongside abra secrets/volumes. +- **Recipe-versioned tests (anti-anchoring).** Per plan §7.1, tests read versions/endpoints + dynamically (the app's own discovery endpoints, env from `live_app`) — never hardcode published + release values. Each functional test file declares the recipe-info SOURCE path it ports from so + the Adversary can audit parity cold. +- **Heavy-recipe parking.** Drone's `MAX_TESTS=1` + per-build timeout already serialize runs; for + Phase 2 we DO NOT lift it. Within a single run, the orchestrator deploys deps before the + recipe-under-test sequentially (never concurrently) per plan §4.2. diff --git a/machine-docs/JOURNAL-2.md b/machine-docs/JOURNAL-2.md new file mode 100644 index 0000000..fdbd027 --- /dev/null +++ b/machine-docs/JOURNAL-2.md @@ -0,0 +1,60 @@ +# JOURNAL — Phase 2 (per-recipe test authoring) + +Builder-private (append-only). Builder rationalisations, dead-ends, in-the-moment reasoning. The +Adversary does NOT read this before forming a verdict; objective evidence goes in STATUS-2 / REVIEW-2. +Phase plan: `/srv/cc-ci/cc-ci-plan/plan-phase2-recipe-tests.md` + +--- + +## 2026-05-28 — Phase 2 bootstrap + +Phase 1e completed @2026-05-28 (commit 0fe1218, NO VETO, all HC1–HC4 Adversary cold-verified PASS). +Foundation is in place: the orchestrator deploys ONCE per run, performs each lifecycle op ONCE +(install→deploy / upgrade→chaos-redeploy of PR head / backup→`abra app backup` / restore→`abra app +restore`), and runs **both** generic (`tests/_generic/test_.py`) and overlay +(`tests//test_.py`) assertion files **additively** against the shared post-op state. +Pre-op seeds live in optional `tests//ops.py` (`pre_install`/`pre_upgrade`/`pre_backup`/ +`pre_restore`). The deploy-count guard (DG4.1) stays =1; teardown is sacred. Per Phase-1e HC1, the +upgrade tier proves PR-head was deployed via `chaos-version` label = `head_ref` (head SHA from +$REF). Per HC2, repo-local PR-authored code runs only for recipes on +`tests/repo-local-approved.txt` (default-deny). + +**Bootstrap (this session):** +1. `git pull --rebase` — already up to date. +2. Verified §1 access: `ssh cc-ci` OK (NixOS 24.11), Gitea API HTTP 200, wildcard + `probe-$RANDOM.ci.commoninternet.net` resolves to gateway `143.244.213.108`. +3. Read the Phase-2 plan + plan.md §6.1/§7/§9 (loop protocol, single-writer ownership, gate + handshake, anti-drift). Read STATUS-1e + REVIEW-1e final to inherit the harness invariants + (HC1–HC4 cold-verified PASS, F1e-2 not blocking). +4. Surveyed existing state: `tests//` already exists for **custom-html, cryptpad, keycloak, + lasuite-docs, matrix-synapse, n8n** — these were built out as Phase-1d/1e overlays + recipe_meta + + ops.py. The lifecycle overlay model (test_install/upgrade/backup/restore.py + ops.py) is the + foundation. Phase 2 adds **parity-port functional tests** + **≥2 NEW recipe-specific tests** + + **dependency/SSO resolver** + **PARITY.md** per recipe. +5. Surveyed `references/recipe-maintainer` (mounted at `/srv/recipe-maintainer/`) — the parity + source. Per-recipe corpus: + - **custom-html** — health_check.py (200 check) + - **n8n** — health_check.py + - **keycloak** — health_check.py + oidc_integration.py (cross-recipe with lasuite-docs) + - **cryptpad** — health_check.py + oidc_login.py + - **lasuite-docs** — health_check.py + oidc_login.py + upload_conversion.py + - **lasuite-meet** — health_check.py + oidc_login.py + meeting_flow.py + webrtc-media.py + + webrtc-relay.py + - **matrix-synapse** — *shell* tests: compress_state.sh + test_complexity_limit.sh + test_purge.sh + (will port semantics to Python under cc-ci) + - **hedgedoc / authentik / immich / bluesky-pds / mumble / gitea / lichen / lichen-markdown** — + no `tests/` dir under recipe-info yet, will fill from plan §4.3 spec. + +**Plan-shape orientation:** +- `tests//test_.py` (lifecycle overlays) — already established. +- `tests//functional/` — Phase-2 introduces this subdir for parity-port + new specific tests. + Discovery currently globs `test_*.py` at the top level only; will need to recurse (Q0.2). +- `tests//playwright/` — same. +- `tests//PARITY.md` — Phase-2 introduces this; mapping table per recipe. + +**Bootstrap commits incoming:** +- Add STATUS-2.md / BACKLOG-2.md / JOURNAL-2.md (this session). +- DECISIONS.md append: PARITY.md format, functional/ + playwright/ subdirs, dep-resolver shape. + +Will now seed DECISIONS, then begin Q0.1 (vendor helpers into runner/harness/) — keeping the +custom-html overlay working as the reference recipe. The /loop will self-pace. diff --git a/machine-docs/STATUS-2.md b/machine-docs/STATUS-2.md new file mode 100644 index 0000000..a30c423 --- /dev/null +++ b/machine-docs/STATUS-2.md @@ -0,0 +1,66 @@ +# STATUS — Phase 2 (per-recipe test authoring) + +**Phase plan (SSOT):** `/srv/cc-ci/cc-ci-plan/plan-phase2-recipe-tests.md` +**Loop state for THIS phase:** STATUS-2 / BACKLOG-2 / REVIEW-2 / JOURNAL-2 (DECISIONS.md shared). +Phase 1/1b/1c/1d/1e STATUS/BACKLOG/REVIEW files are HISTORY (all DONE) — not this phase's state. + +## Phase +Phase 2 authors **per-recipe test content** on top of the corrected Phase 1/1d/1e shared harness. +Per the plan, for every maintained Co-op Cloud recipe (§5 target set), the cc-ci `tests//` +tree must carry: +- Phase-1d/1e **lifecycle overlays** (assertion-only, additive) — `test_install.py`, `test_upgrade.py`, + `test_backup.py`, `test_restore.py` + `ops.py` pre-op seeds. +- **Parity-ported** tests from `references/recipe-maintainer/recipe-info//tests/*.py`, + one-to-one (P2), with a `PARITY.md` mapping table. +- **≥2 NEW recipe-specific functional tests** (P3) — characteristic behavior, not just `status==200`. +- **Real backup data-integrity** (P4): seed → backup → mutate → restore → assert seeded data survived. +- **Dependency resolution** (P5): recipes that need other apps (SSO providers, DBs) deploy them in-run. +- **Playwright** (P6) where the app's core UX is a UI flow. +- **Docs** (P8): `docs/enroll-recipe.md` updated with the per-recipe test contract + worked example. + +## Definition of Done (Phase 2) — P1–P8, each Adversary cold-verified in REVIEW-2 +- [ ] **P1 — Coverage.** Every recipe in §5 target set has a `tests//` suite enrolled and a + full green `!testme` run (install + upgrade + backup-restore). +- [ ] **P2 — Parity port.** Every `recipe-info//tests/*.py` has a comparable cc-ci test; + `tests//PARITY.md` records the mapping; non-ports documented in DECISIONS.md. +- [ ] **P3 — Recipe-specific depth.** Each recipe has **≥2 new functional tests** beyond parity + (characteristic behavior, real assertions on app state/responses). +- [ ] **P4 — Backup data-integrity is real.** Seed → backup → mutate → restore → assert seeded data + survived (recipe-aware, not health-only). Pattern already proven in Phase 1e on custom-html. +- [ ] **P5 — Dependencies handled.** Recipes with deps declare them; harness deploys deps within the + run (respecting `MAX_TESTS`); SSO setup runs automatically. +- [ ] **P6 — Browser flows where they matter (D3).** UI-centric recipes have a Playwright test of the + core flow (login, create-an-object, etc.). +- [ ] **P7 — No weakened tests, no corners cut.** Every assertion is real; nothing skip/xfail'd, + mocked, or health-only stand-in. Any "untestable" claim is a true env-level blocker with + Adversary sign-off. +- [ ] **P8 — Docs.** `docs/enroll-recipe.md` updated with the per-recipe test contract (§4.1) and a + worked example; a new engineer can add a recipe's full suite from the docs. + +## Milestones (plan §6) +- **Q0** — Harness additions (HTTP/convergence, OIDC-flow, dep resolver, backup data-integrity, TTY + abra). Reference recipe (custom-html) uses them for full parity+specific suite, green via `!testme`. +- **Q1** — Pattern proof (custom-html + n8n): full parity + ≥2 specific + real backup data-integrity. +- **Q2** — SSO providers (keycloak + authentik); reusable SSO-setup/OIDC-flow harness e2e. +- **Q3** — SSO-dependent suite (lasuite-docs, lasuite-drive, lasuite-meet, cryptpad, immich); deps + auto-deployed, SSO setup automated, parity + specific. +- **Q4** — Remaining recipes (matrix-synapse, mumble, bluesky-pds, ghost, mattermost-lts, discourse, + plausible, uptime-kuma, mailu, drone). +- **Q5** — Completeness + docs; flip `## DONE`. + +## In flight +**Q0 — Harness additions.** Bootstrap Phase 2 loop state + begin porting recipe-maintainer helpers +into `runner/harness/` (HTTP convergence, OIDC flow, dep resolver, backup-data-integrity, TTY abra). + +## Gate +(none yet — Q0 has not been claimed) + +## Blocked +(none) — bootstrap access re-verified @2026-05-28: `ssh cc-ci` ok (root, NixOS 24.11), Gitea API +HTTP 200, wildcard DNS resolves to gateway 143.244.213.108. + +## Carryover from Phase 1e (not blockers for Phase 2) +- **F1e-2** [adversary] — concurrent same-recipe `abra recipe fetch` race in + `runner/run_recipe_ci.py::fetch_recipe`. Pre-existing in Phase 1d; not a 1e regression. Drone + caps `MAX_TESTS=1` today, so practical impact bounded. Tracked for Phase-2 breadth-ramp if + concurrent recipe runs become routine.