chore(2): bootstrap Phase 2 loop state + decisions

- STATUS-2.md / BACKLOG-2.md / JOURNAL-2.md seeded from plan §6 (Q0-Q5).
- DECISIONS.md appended Phase 2 section: functional/ + playwright/ subdirs,
  PARITY.md mapping convention, vendored helpers in runner/harness/
  (http, abra_tty, deps, sso, data_integrity), recipe-versioned tests.
- Bootstrap access re-verified: ssh cc-ci ok, Gitea API 200, wildcard DNS to
  gateway 143.244.213.108.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

machine-docs/BACKLOG-2.md

@@ -0,0 +1,88 @@
+# BACKLOG — Phase 2 (per-recipe test authoring)
+
+Phase-namespaced backlog. Builder edits `## Build backlog`; Adversary edits `## Adversary findings`.
+Phase plan: `/srv/cc-ci/cc-ci-plan/plan-phase2-recipe-tests.md`
+
+## Build backlog
+
+### Q0 — Harness additions
+- [ ] **Q0.1** — Vendor `recipe-maintainer/utils/tests/helpers.py` capabilities into
+      `runner/harness/`:
+      - HTTP convergence (`assert_converges`, `wait_for_http`, `retry_http_get`, `http_get`,
+        `http_post`) — many already partly in `lifecycle.py`/`generic.py`; add the missing pieces +
+        consolidate (no duplicate logic).
+      - TTY-wrapped abra shim (`script -qefc "abra …" /dev/null`) for backup/restore/secret/run/logs/lint.
+      - Backup data-integrity primitive (volume/DB seed → marker survival check).
+- [ ] **Q0.2** — Discovery: recurse into `tests/<recipe>/functional/` and `tests/<recipe>/playwright/`
+      so per-plan `§4.1` directory layout is honored. Update `discovery.custom_tests` + `enroll-recipe.md`.
+- [ ] **Q0.3** — PARITY.md template + a worked custom-html example
+      (`tests/custom-html/PARITY.md`). Port custom-html health_check.py to a functional test.
+- [ ] **Q0.4** — Dependency resolver harness primitive (read `tests/<recipe>/recipe.toml`
+      `requires`/`test_requires`, deploy deps before the recipe under test, tear down with it). Mind
+      `MAX_TESTS`/node budget; sequence heavy ones. Defer SSO-realm-setup wiring to Q2.
+- [ ] **Q0.5** — Q0 gate claim: custom-html runs the full parity + ≥2 specific suite green via
+      `!testme`; PARITY.md complete for custom-html; harness changes recorded in DECISIONS.md.
+
+### Q1 — Pattern proof (custom-html + n8n)
+- [ ] **Q1.1** — custom-html: ≥2 NEW recipe-specific functional tests (beyond parity
+      `health_check`). Candidates from plan §4.3: "serve/persist content: write content, fetch it back"
+      — custom-html serves the `/usr/share/nginx/html` volume, so a content round-trip + a content-type
+      header check are appropriate. Playwright already exists in `tests/custom-html/test_install.py`.
+- [ ] **Q1.2** — n8n: enroll under cc-ci. Port `recipe-info/n8n/tests/health_check.py` →
+      `tests/n8n/functional/health_check.py`. Add ≥2 specific tests: (a) create a workflow via API,
+      execute it, assert result; (b) the workflow persists across an upgrade. PARITY.md filled.
+- [ ] **Q1.3** — Real backup data-integrity for n8n: seed a workflow → backup → wipe → restore → list
+      workflows, prove the seeded one survived. (custom-html already has this pattern from 1e.)
+- [ ] **Q1.4** — Q1 gate: both recipes green via `!testme`; both PARITY.md complete.
+
+### Q2 — SSO providers (keycloak + authentik)
+- [ ] **Q2.1** — keycloak: port `tests/keycloak/oidc_integration.py` (the dependent-recipe test) and
+      `tests/health_check.py`. Add specific tests from plan §4.3 (realm+client via admin API; password
+      and client-credentials token grants; JWT claims).
+- [ ] **Q2.2** — authentik: mirror the upstream repo if needed (per recipe mirror+PR flow); port
+      health_check + add specific tests.
+- [ ] **Q2.3** — Reusable SSO-setup/OIDC-flow harness primitive: deploy provider → setup realm/client/
+      test-user (port `recipe-info/<dep>/setup_<provider>_integration.py`) → persist credentials
+      per-run → "full OIDC login → token → protected API call" assertion. Implement once in
+      `runner/harness/`; reused by every SSO-dependent recipe.
+- [ ] **Q2.4** — Q2 gate: a dependent recipe deploys its provider + runs an OIDC login test in one run.
+
+### Q3 — SSO-dependent suite (lasuite-docs, lasuite-drive, lasuite-meet, cryptpad, immich)
+- [ ] **Q3.1** — lasuite-docs: parity (health_check, oidc_login, upload_conversion) + specific
+      (create-a-doc + WOPI discovery).
+- [ ] **Q3.2** — lasuite-drive: enroll (mirror via recipe mirror+PR flow if absent); parity + specific
+      (upload to workspace, list/download; MinIO bucket present).
+- [ ] **Q3.3** — lasuite-meet: parity (health_check, oidc_login, meeting_flow, webrtc-media,
+      webrtc-relay) + specific (create-a-room, two-user LiveKit token issuance, ICE-candidate gathering).
+- [ ] **Q3.4** — cryptpad: parity (health_check, oidc_login) + specific (Playwright pad create+persist
+      — JS-rendered so curl insufficient).
+- [ ] **Q3.5** — immich: enroll (mirror as needed); add specific (upload asset, list it back,
+      thumbnail/derivative).
+- [ ] **Q3.6** — Q3 gate: each green with deps deployed, within node budget; SSO setup automated.
+
+### Q4 — Remaining recipes
+- [ ] **Q4.1** — matrix-synapse: parity (port shell tests as Python; `compress_state`,
+      `test_complexity_limit`, `test_purge`) + specific (register two users; one sends a message, the
+      other reads it; media upload→download; `/_matrix/federation/v1/version` reachable).
+- [ ] **Q4.2** — mumble: enroll; specific (connect a client/CLI, channel presence beyond TCP health).
+- [ ] **Q4.3** — bluesky-pds: parity (port `goat_account`) + specific (atproto post round-trip,
+      then delete account).
+- [ ] **Q4.4** — ghost: enroll; specific (create-a-post round-trip).
+- [ ] **Q4.5** — mattermost-lts: enroll; specific (create-a-message round-trip).
+- [ ] **Q4.6** — discourse: enroll; specific (create-a-topic round-trip).
+- [ ] **Q4.7** — plausible: enroll; specific (track a test event, query it back).
+- [ ] **Q4.8** — uptime-kuma: enroll; specific (create a monitor, list it).
+- [ ] **Q4.9** — mailu: enroll; specific (create a mailbox, send/receive verification).
+- [ ] **Q4.10** — drone: enroll; specific (create/list builds via API).
+- [ ] **Q4.11** — Q4 gate: each recipe green with parity + specific.
+
+### Q5 — Completeness + docs
+- [ ] **Q5.1** — `docs/enroll-recipe.md` updated with the per-recipe test contract (§4.1), the
+      `functional/` and `playwright/` subdirectory layout, the PARITY.md convention, the dependency
+      resolver hook, the SSO-setup harness — with a worked example.
+- [ ] **Q5.2** — Adversary samples a subset and cold-verifies parity tables + specific tests are real
+      (not health-only, not skipped). NO weakened test, no corners cut (P7).
+- [ ] **Q5.3** — Phase 2 `## DONE` after all P1–P8 Adversary cold-verified PASS, no standing VETO.
+
+## Adversary findings
+(empty — Adversary writes here)

machine-docs/DECISIONS.md

@@ -379,3 +379,64 @@ Three operator-review corrections to the Phase-1d shared harness, settled here (
   `coop-cloud.<stack>.chaos`/`.chaos-version`) — the chaos label IS the proof PR-head was deployed.
   Non-PR `!testme` (no SRC/REF): "PR head" = the catalogue current checkout, so upgrade is prev→current
   — still a genuine move via chaos. (Exact chaos label name verified on the live abra during E2.)
+
+## Phase 2 — per-recipe test authoring (design, 2026-05-28)
+Inherits the Phase 1d/1e shared-deployment + additive-overlay + op/assertion-split model. Phase 2
+adds **content**, not infra, with a few small harness primitives ported from
+`references/recipe-maintainer/utils/tests/helpers.py`.
+
+- **Per-recipe layout (per plan §4.1).** The cc-ci `tests/<recipe>/` dir continues to use the
+  Phase-1d/1e overlays at the top level (`test_install.py`, `test_upgrade.py`, `test_backup.py`,
+  `test_restore.py`, `ops.py`, `recipe_meta.py`, optional `install_steps.sh`). NEW Phase-2
+  subdirectories:
+  - `tests/<recipe>/functional/` — parity-port tests (one per recipe-maintainer `tests/*.py`) +
+    ≥2 NEW recipe-specific functional tests (P2/P3). Each file is `test_*.py` (pytest-discoverable);
+    each parity port carries a **`SOURCE = "recipe-info/<recipe>/tests/<file>"`** comment near the
+    top so the audit trail is in the file, not just in PARITY.md.
+  - `tests/<recipe>/playwright/` — browser flows (P6) where the app's UX is a UI flow. Same
+    `test_*.py` convention; each file imports `playwright.sync_api`.
+  - `tests/<recipe>/PARITY.md` — required mapping table (P2) with one row per
+    recipe-info parity test: `| recipe-maintainer file | cc-ci file | what's verified | status |`.
+    A deliberate non-port is a documented row in DECISIONS.md (linked from PARITY.md), not a silent
+    omission.
+- **Discovery for the new subdirs.** `runner/harness/discovery.custom_tests` recurses into
+  `tests/<recipe>/functional/` and `tests/<recipe>/playwright/` (in addition to the top-level glob),
+  so Phase-2 functional tests run as part of the **custom** stage automatically. Repo-local (HC2)
+  gate still applies if the recipe is approved; otherwise only cc-ci's own functional/ + playwright/
+  run. The top-level `test_install.py`/etc. continue to drive the lifecycle overlays — the
+  `functional/` + `playwright/` files are **always custom-stage**, never lifecycle (so they don't
+  perform an op; they assert against the post-install live deployment).
+- **Vendored helpers in `runner/harness/`.** Capabilities ported from `recipe-maintainer/utils/tests/
+  helpers.py` (cc-ci is self-contained at runtime — does NOT import recipe-maintainer's workspace,
+  per plan §8 default):
+  - `harness.http` — `http_get(url, headers=, timeout=) -> (status, json_or_None)`,
+    `http_post(...)`, `retry_http_get(url, timeout=, **)`, `wait_for_http(url, label, max_wait=)`,
+    `assert_converges(fn, description, max_wait=, interval=)`. (Several variants exist
+    `lifecycle.http_fetch/http_get/http_body` already; the harness.http module is the **canonical**
+    Phase-2 HTTP API for tests; lifecycle.* helpers stay for infra-level checks.)
+  - `harness.abra_tty` — `script -qefc "abra …" /dev/null` wrapper for the abra commands that
+    require a TTY (backup/restore/secret/run/logs/lint), used by parity tests that drive abra
+    directly. Lifecycle already exposes typed wrappers — this is for tests that need raw shell-abra.
+  - `harness.deps` — dependency resolver primitive. Reads `tests/<recipe>/recipe.toml`
+    (`requires` / `test_requires`), deploys each declared dep via the same `lifecycle.deploy_app`
+    + `wait_healthy` path (so the dep is a real `<dep[:4]>-<6hex>.ci.commoninternet.net` on the
+    same swarm), persists per-run, tears down with the parent in the orchestrator's `finally`.
+    Heavy recipes sequence sequentially; `MAX_TESTS`/node budget is the cap.
+  - `harness.sso` — OIDC-flow primitive (Q2 deliverable). Given a deployed provider domain and a
+    recipe-defined realm/client/test-user, performs the full "deploy provider → setup realm/client
+    via admin API → obtain access token (password + client-credentials grants) → assert protected
+    API call accepts it" assertion. Reusable by every SSO-dependent recipe (cryptpad, lasuite-*,
+    immich, etc.). Setup scripts ported from `recipe-info/<dep>/setup_<provider>_integration.py`.
+  - `harness.data_integrity` — backup data-integrity primitive: a recipe-aware "seed a marker
+    →  backup → mutate → restore → assert seeded marker survived" helper around `lifecycle.exec_in_app`
+    / `http_get` (the recipe chooses the marker mechanism, the helper guarantees the pattern).
+- **Run-scoped credentials for SSO/recipe-specific tests** (plan §4.4 class-B). Generated secrets
+  (realm/client/test-user passwords, API tokens) persist for the run via the existing
+  `runs/<app-name>/` mechanism (Phase 1d). Destroyed at teardown alongside abra secrets/volumes.
+- **Recipe-versioned tests (anti-anchoring).** Per plan §7.1, tests read versions/endpoints
+  dynamically (the app's own discovery endpoints, env from `live_app`) — never hardcode published
+  release values. Each functional test file declares the recipe-info SOURCE path it ports from so
+  the Adversary can audit parity cold.
+- **Heavy-recipe parking.** Drone's `MAX_TESTS=1` + per-build timeout already serialize runs; for
+  Phase 2 we DO NOT lift it. Within a single run, the orchestrator deploys deps before the
+  recipe-under-test sequentially (never concurrently) per plan §4.2.

machine-docs/JOURNAL-2.md

@@ -0,0 +1,60 @@
+# JOURNAL — Phase 2 (per-recipe test authoring)
+
+Builder-private (append-only). Builder rationalisations, dead-ends, in-the-moment reasoning. The
+Adversary does NOT read this before forming a verdict; objective evidence goes in STATUS-2 / REVIEW-2.
+Phase plan: `/srv/cc-ci/cc-ci-plan/plan-phase2-recipe-tests.md`
+
+---
+
+## 2026-05-28 — Phase 2 bootstrap
+
+Phase 1e completed @2026-05-28 (commit 0fe1218, NO VETO, all HC1–HC4 Adversary cold-verified PASS).
+Foundation is in place: the orchestrator deploys ONCE per run, performs each lifecycle op ONCE
+(install→deploy / upgrade→chaos-redeploy of PR head / backup→`abra app backup` / restore→`abra app
+restore`), and runs **both** generic (`tests/_generic/test_<op>.py`) and overlay
+(`tests/<recipe>/test_<op>.py`) assertion files **additively** against the shared post-op state.
+Pre-op seeds live in optional `tests/<recipe>/ops.py` (`pre_install`/`pre_upgrade`/`pre_backup`/
+`pre_restore`). The deploy-count guard (DG4.1) stays =1; teardown is sacred. Per Phase-1e HC1, the
+upgrade tier proves PR-head was deployed via `chaos-version` label = `head_ref` (head SHA from
+$REF). Per HC2, repo-local PR-authored code runs only for recipes on
+`tests/repo-local-approved.txt` (default-deny).
+
+**Bootstrap (this session):**
+1. `git pull --rebase` — already up to date.
+2. Verified §1 access: `ssh cc-ci` OK (NixOS 24.11), Gitea API HTTP 200, wildcard
+   `probe-$RANDOM.ci.commoninternet.net` resolves to gateway `143.244.213.108`.
+3. Read the Phase-2 plan + plan.md §6.1/§7/§9 (loop protocol, single-writer ownership, gate
+   handshake, anti-drift). Read STATUS-1e + REVIEW-1e final to inherit the harness invariants
+   (HC1–HC4 cold-verified PASS, F1e-2 not blocking).
+4. Surveyed existing state: `tests/<recipe>/` already exists for **custom-html, cryptpad, keycloak,
+   lasuite-docs, matrix-synapse, n8n** — these were built out as Phase-1d/1e overlays + recipe_meta
+   + ops.py. The lifecycle overlay model (test_install/upgrade/backup/restore.py + ops.py) is the
+   foundation. Phase 2 adds **parity-port functional tests** + **≥2 NEW recipe-specific tests** +
+   **dependency/SSO resolver** + **PARITY.md** per recipe.
+5. Surveyed `references/recipe-maintainer` (mounted at `/srv/recipe-maintainer/`) — the parity
+   source. Per-recipe corpus:
+   - **custom-html** — health_check.py (200 check)
+   - **n8n** — health_check.py
+   - **keycloak** — health_check.py + oidc_integration.py (cross-recipe with lasuite-docs)
+   - **cryptpad** — health_check.py + oidc_login.py
+   - **lasuite-docs** — health_check.py + oidc_login.py + upload_conversion.py
+   - **lasuite-meet** — health_check.py + oidc_login.py + meeting_flow.py + webrtc-media.py +
+     webrtc-relay.py
+   - **matrix-synapse** — *shell* tests: compress_state.sh + test_complexity_limit.sh + test_purge.sh
+     (will port semantics to Python under cc-ci)
+   - **hedgedoc / authentik / immich / bluesky-pds / mumble / gitea / lichen / lichen-markdown** —
+     no `tests/` dir under recipe-info yet, will fill from plan §4.3 spec.
+
+**Plan-shape orientation:**
+- `tests/<recipe>/test_<op>.py` (lifecycle overlays) — already established.
+- `tests/<recipe>/functional/` — Phase-2 introduces this subdir for parity-port + new specific tests.
+  Discovery currently globs `test_*.py` at the top level only; will need to recurse (Q0.2).
+- `tests/<recipe>/playwright/` — same.
+- `tests/<recipe>/PARITY.md` — Phase-2 introduces this; mapping table per recipe.
+
+**Bootstrap commits incoming:**
+- Add STATUS-2.md / BACKLOG-2.md / JOURNAL-2.md (this session).
+- DECISIONS.md append: PARITY.md format, functional/ + playwright/ subdirs, dep-resolver shape.
+
+Will now seed DECISIONS, then begin Q0.1 (vendor helpers into runner/harness/) — keeping the
+custom-html overlay working as the reference recipe. The /loop will self-pace.

machine-docs/STATUS-2.md

@@ -0,0 +1,66 @@
+# STATUS — Phase 2 (per-recipe test authoring)
+
+**Phase plan (SSOT):** `/srv/cc-ci/cc-ci-plan/plan-phase2-recipe-tests.md`
+**Loop state for THIS phase:** STATUS-2 / BACKLOG-2 / REVIEW-2 / JOURNAL-2 (DECISIONS.md shared).
+Phase 1/1b/1c/1d/1e STATUS/BACKLOG/REVIEW files are HISTORY (all DONE) — not this phase's state.
+
+## Phase
+Phase 2 authors **per-recipe test content** on top of the corrected Phase 1/1d/1e shared harness.
+Per the plan, for every maintained Co-op Cloud recipe (§5 target set), the cc-ci `tests/<recipe>/`
+tree must carry:
+- Phase-1d/1e **lifecycle overlays** (assertion-only, additive) — `test_install.py`, `test_upgrade.py`,
+  `test_backup.py`, `test_restore.py` + `ops.py` pre-op seeds.
+- **Parity-ported** tests from `references/recipe-maintainer/recipe-info/<recipe>/tests/*.py`,
+  one-to-one (P2), with a `PARITY.md` mapping table.
+- **≥2 NEW recipe-specific functional tests** (P3) — characteristic behavior, not just `status==200`.
+- **Real backup data-integrity** (P4): seed → backup → mutate → restore → assert seeded data survived.
+- **Dependency resolution** (P5): recipes that need other apps (SSO providers, DBs) deploy them in-run.
+- **Playwright** (P6) where the app's core UX is a UI flow.
+- **Docs** (P8): `docs/enroll-recipe.md` updated with the per-recipe test contract + worked example.
+
+## Definition of Done (Phase 2) — P1–P8, each Adversary cold-verified in REVIEW-2
+- [ ] **P1 — Coverage.** Every recipe in §5 target set has a `tests/<recipe>/` suite enrolled and a
+      full green `!testme` run (install + upgrade + backup-restore).
+- [ ] **P2 — Parity port.** Every `recipe-info/<recipe>/tests/*.py` has a comparable cc-ci test;
+      `tests/<recipe>/PARITY.md` records the mapping; non-ports documented in DECISIONS.md.
+- [ ] **P3 — Recipe-specific depth.** Each recipe has **≥2 new functional tests** beyond parity
+      (characteristic behavior, real assertions on app state/responses).
+- [ ] **P4 — Backup data-integrity is real.** Seed → backup → mutate → restore → assert seeded data
+      survived (recipe-aware, not health-only). Pattern already proven in Phase 1e on custom-html.
+- [ ] **P5 — Dependencies handled.** Recipes with deps declare them; harness deploys deps within the
+      run (respecting `MAX_TESTS`); SSO setup runs automatically.
+- [ ] **P6 — Browser flows where they matter (D3).** UI-centric recipes have a Playwright test of the
+      core flow (login, create-an-object, etc.).
+- [ ] **P7 — No weakened tests, no corners cut.** Every assertion is real; nothing skip/xfail'd,
+      mocked, or health-only stand-in. Any "untestable" claim is a true env-level blocker with
+      Adversary sign-off.
+- [ ] **P8 — Docs.** `docs/enroll-recipe.md` updated with the per-recipe test contract (§4.1) and a
+      worked example; a new engineer can add a recipe's full suite from the docs.
+
+## Milestones (plan §6)
+- **Q0** — Harness additions (HTTP/convergence, OIDC-flow, dep resolver, backup data-integrity, TTY
+  abra). Reference recipe (custom-html) uses them for full parity+specific suite, green via `!testme`.
+- **Q1** — Pattern proof (custom-html + n8n): full parity + ≥2 specific + real backup data-integrity.
+- **Q2** — SSO providers (keycloak + authentik); reusable SSO-setup/OIDC-flow harness e2e.
+- **Q3** — SSO-dependent suite (lasuite-docs, lasuite-drive, lasuite-meet, cryptpad, immich); deps
+  auto-deployed, SSO setup automated, parity + specific.
+- **Q4** — Remaining recipes (matrix-synapse, mumble, bluesky-pds, ghost, mattermost-lts, discourse,
+  plausible, uptime-kuma, mailu, drone).
+- **Q5** — Completeness + docs; flip `## DONE`.
+
+## In flight
+**Q0 — Harness additions.** Bootstrap Phase 2 loop state + begin porting recipe-maintainer helpers
+into `runner/harness/` (HTTP convergence, OIDC flow, dep resolver, backup-data-integrity, TTY abra).
+
+## Gate
+(none yet — Q0 has not been claimed)
+
+## Blocked
+(none) — bootstrap access re-verified @2026-05-28: `ssh cc-ci` ok (root, NixOS 24.11), Gitea API
+HTTP 200, wildcard DNS resolves to gateway 143.244.213.108.
+
+## Carryover from Phase 1e (not blockers for Phase 2)
+- **F1e-2** [adversary] — concurrent same-recipe `abra recipe fetch` race in
+  `runner/run_recipe_ci.py::fetch_recipe`. Pre-existing in Phase 1d; not a 1e regression. Drone
+  caps `MAX_TESTS=1` today, so practical impact bounded. Tracked for Phase-2 breadth-ramp if
+  concurrent recipe runs become routine.