cc-ci/machine-docs/STATUS-2.md

# STATUS — Phase 2 (per-recipe test authoring)

**Phase plan (SSOT):** `/srv/cc-ci/cc-ci-plan/plan-phase2-recipe-tests.md`
**Loop state for THIS phase:** STATUS-2 / BACKLOG-2 / REVIEW-2 / JOURNAL-2 (DECISIONS.md shared).
Phase 1/1b/1c/1d/1e STATUS/BACKLOG/REVIEW files are HISTORY (all DONE) — not this phase's state.

## Phase
Phase 2 authors **per-recipe test content** on top of the corrected Phase 1/1d/1e shared harness.
Per the plan, for every maintained Co-op Cloud recipe (§5 target set), the cc-ci `tests/<recipe>/`
tree must carry:
- Phase-1d/1e **lifecycle overlays** (assertion-only, additive) — `test_install.py`, `test_upgrade.py`,
  `test_backup.py`, `test_restore.py` + `ops.py` pre-op seeds.
- **Parity-ported** tests from `references/recipe-maintainer/recipe-info/<recipe>/tests/*.py`,
  one-to-one (P2), with a `PARITY.md` mapping table.
- **≥2 NEW recipe-specific functional tests** (P3) — characteristic behavior, not just `status==200`.
- **Real backup data-integrity** (P4): seed → backup → mutate → restore → assert seeded data survived.
- **Dependency resolution** (P5): recipes that need other apps (SSO providers, DBs) deploy them in-run.
- **Playwright** (P6) where the app's core UX is a UI flow.
- **Docs** (P8): `docs/enroll-recipe.md` updated with the per-recipe test contract + worked example.

## Definition of Done (Phase 2) — P1–P8, each Adversary cold-verified in REVIEW-2
- [ ] **P1 — Coverage.** Every recipe in §5 target set has a `tests/<recipe>/` suite enrolled and a
      full green `!testme` run (install + upgrade + backup-restore).
- [ ] **P2 — Parity port.** Every `recipe-info/<recipe>/tests/*.py` has a comparable cc-ci test;
      `tests/<recipe>/PARITY.md` records the mapping; non-ports documented in DECISIONS.md.
- [ ] **P3 — Recipe-specific depth.** Each recipe has **≥2 new functional tests** beyond parity
      (characteristic behavior, real assertions on app state/responses).
- [ ] **P4 — Backup data-integrity is real.** Seed → backup → mutate → restore → assert seeded data
      survived (recipe-aware, not health-only). Pattern already proven in Phase 1e on custom-html.
- [ ] **P5 — Dependencies handled.** Recipes with deps declare them; harness deploys deps within the
      run (respecting `MAX_TESTS`); SSO setup runs automatically.
- [ ] **P6 — Browser flows where they matter (D3).** UI-centric recipes have a Playwright test of the
      core flow (login, create-an-object, etc.).
- [ ] **P7 — No weakened tests, no corners cut.** Every assertion is real; nothing skip/xfail'd,
      mocked, or health-only stand-in. Any "untestable" claim is a true env-level blocker with
      Adversary sign-off.
- [ ] **P8 — Docs.** `docs/enroll-recipe.md` updated with the per-recipe test contract (§4.1) and a
      worked example; a new engineer can add a recipe's full suite from the docs.

## Milestones (plan §6)
- **Q0** — Harness additions (HTTP/convergence, OIDC-flow, dep resolver, backup data-integrity, TTY
  abra). Reference recipe (custom-html) uses them for full parity+specific suite, green via `!testme`.
- **Q1** — Pattern proof (custom-html + n8n): full parity + ≥2 specific + real backup data-integrity.
- **Q2** — SSO providers (keycloak + authentik); reusable SSO-setup/OIDC-flow harness e2e.
- **Q3** — SSO-dependent suite (lasuite-docs, lasuite-drive, lasuite-meet, cryptpad, immich); deps
  auto-deployed, SSO setup automated, parity + specific.
- **Q4** — Remaining recipes (matrix-synapse, mumble, bluesky-pds, ghost, mattermost-lts, discourse,
  plausible, uptime-kuma, mailu, drone).
- **Q5** — Completeness + docs; flip `## DONE`.

## In flight
**Q2 — SSO providers (keycloak + authentik).** Q1 CLAIMED — n8n + custom-html both full-green via
the existing run path; PARITY.md + functional/ + playwright/ in place; data-integrity proven via
the lifecycle overlay pattern. Next: keycloak parity port + OIDC-flow harness primitive.

## Gate
**Gate: Q1 — RE-CLAIMED, awaiting Adversary @2026-05-28** (commit `fc89552` F2-3 + F2-4 fixes on
top of original Q1 claim `2f3d5aa`). Acceptance per plan §6 Q1 + Adversary Q1 FAIL findings:
- **F2-4 closed:** `tests/n8n/functional/test_workflow_roundtrip.py` lands the plan §4.3
  prescribed test (owner setup → workflow create → workflow read-back), removing the prohibited
  "needs owner setup" excuse. Cold e2e PASS.
- **F2-3 closed:** `tests/n8n/test_install.py` now wraps `page.goto(...)` in `try/except
  PlaywrightError` inside the retry loop so transient `net::ERR_*` errors trigger a retry instead
  of test failure. Cold e2e PASS first try.

**Objective evidence pointers (Q1):**
- **custom-html (Q1.1)** — already cold-verified in Q0 PASS. Same evidence stands: full e2e green,
  HC1 non-vacuous, deploy-count=1; PARITY.md + functional/ + playwright/ in place.
- **n8n (Q1.2)** — full e2e on cc-ci (log `/root/ccci-q1-n8n-r3.log`):
  - **HC1 PR-head proof:** `head_ref=63dd3e0f == chaos-version=63dd3e0f`, version
    `3.1.0+2.9.4 → 3.2.0+2.20.6`.
  - **Deploy-count = 1** (DG4.1 holds).
  - **Lifecycle tier results (generic + cc-ci overlay both PASS at each stage):**
    - install: generic `test_serving` PASS + cc-ci `test_serving_and_editor` PASS (the robust
      Playwright poll handles n8n's /healthz-200-before-/-route-registered window).
    - upgrade: generic `test_upgrade_reconverges` PASS + cc-ci `test_upgrade_preserves_data`
      PASS (marker `upgrade-survives` written into /home/node/.n8n by `ops.pre_upgrade` survived
      the chaos redeploy of PR-head).
    - backup: generic `test_backup_artifact` PASS + cc-ci `test_backup_captures_state` PASS
      (marker `original` from `ops.pre_backup` captured by `abra app backup create`).
    - restore: generic `test_restore_healthy` PASS + cc-ci `test_restore_returns_state` PASS
      (marker mutated to `mutated` by `ops.pre_restore`, restored to `original` — real backup
      data-integrity).
  - **Custom tier results (4 PASS — log `/root/ccci-q1-n8n-r4.log` post-F2-4/F2-3 fix):**
    - `tests/n8n/functional/test_health_check.py::test_n8n_returns_200` — parity port (HTTP 200
      from `/`), with `SOURCE: recipe-info/n8n/tests/health_check.py` comment.
    - `tests/n8n/functional/test_workflow_roundtrip.py::test_workflow_create_and_read_back` —
      **plan §4.3 prescribed create+read-back**: owner setup → POST /rest/workflows → GET
      /rest/workflows/<id>; assert id/name/nodes round-trip. (F2-4 fix.)
    - `tests/n8n/functional/test_rest_settings.py::test_rest_settings_returns_json_with_known_keys`
      — polls `/rest/settings` until content-type is `application/json` (rejecting the
      "n8n is starting up" placeholder HTML), then asserts known public-settings keys
      (`userManagement` / `defaultLocale` / `authCookie`) in the `data` envelope.
    - `tests/n8n/functional/test_login_state.py::test_login_endpoint_returns_json` — polls
      `/rest/login` until content-type is `application/json`, proves auth subsystem initialized.
  - **PARITY.md complete:** `tests/n8n/PARITY.md` — parity row for `health_check.py`, rationale
    for the 2 recipe-specific tests, data-integrity + playwright sections.
- **Q1 has no Adversary findings yet.** No tests skipped/weakened; rejecting-the-placeholder
  pattern in the new functional tests is non-vacuous (a stuck-booting n8n that only serves the
  placeholder fails the test).

**Reference command for Adversary (cold, on cc-ci):**
```
ssh cc-ci 'cd /root/<your-clone> && RECIPE=n8n cc-ci-run runner/run_recipe_ci.py'
```

---

**Gate: Q0 — Adversary PASS @2026-05-28** (REVIEW-2 `## Q0 — PASS @2026-05-28`; cold re-verify on
`/root/adv-verify` HEAD `0b834e9` → 21 unit PASS + e2e PASS; NO VETO). F2-1 closed; F2-2 (scope
observation) acknowledged.

**Prior Q0 claim detail (commit `5741e88` — F2-1 fix landed on top of the original Q0 changeset).** Acceptance evidence (per plan §6 Q0): a reference recipe
(custom-html) uses the new harness additions for a full parity + specific suite, green via the
existing run path. F2-1 (test_custom_tests_repo_local_gated stale assertion) closed by Builder; cold
re-run on cc-ci → **21/21 PASS** including the previously-failing test. F2-2 (scope observation:
OIDC-flow + dep resolver not in Q0) acknowledged — those primitives implement when Q2/Q3 consume
them; BACKLOG-2 Q0.4 remains open and explicitly deferred.

**Objective evidence pointers (Q0):**
- **Harness additions landed**
  - `runner/harness/http.py` — canonical Phase-2 recipe-test HTTP API (vendored from
    `references/recipe-maintainer/utils/tests/helpers.py`): `http_get`, `http_post`, `http_request`,
    `retry_http_get`, `retry_http_post`, `wait_for_http`, `assert_converges`. JSON + form bodies,
    transport-failure → status=0.
  - `runner/harness/discovery.custom_tests` recurses into `tests/<recipe>/functional/` and
    `tests/<recipe>/playwright/` (Phase 2 §4.1 layout) while excluding lifecycle `test_<op>.py`
    names; HC2 repo-local gate continues to apply.
  - TTY abra wrapper already present in `runner/harness/abra.py::_run_pty` (Phase 1d) — reused.
- **Unit-test proof (deterministic, cc-ci; post-F2-1 fix commit `5741e88`)**
  - `cc-ci-run -m pytest tests/unit -v` → **21 passed in 5.38s** (the previously-failing
    `test_custom_tests_repo_local_gated` now passes; synthetic-recipe + monkeypatch fixture):
    - 8× pre-existing `tests/unit/test_discovery.py` (overlay + HC2 gate, regressed).
    - 2× new `tests/unit/test_discovery_phase2.py` (functional/+playwright/ recursion + HC2
      gate still applies to subdirs).
    - 11× new `tests/unit/test_http.py` (in-process http.server fixture — JSON parsing,
      4xx-with-body, non-JSON body, transport-failure=0, headers, JSON+form POST, retry
      convergence, retry timeout, wait_for_http, assert_converges return value).
- **End-to-end proof (custom-html on cc-ci, the reference recipe)**
  - `RECIPE=custom-html cc-ci-run runner/run_recipe_ci.py` (log `/root/ccci-q0-customhtml-full.log`):
    - install/upgrade/backup/restore/custom **all PASS**, deploy-count=1.
    - HC1 PR-head proof: `head_ref=8a026066 == chaos-version=8a026066`, version `1.10.0→1.11.0`.
    - 5 lifecycle assertions (generic + cc-ci overlay across 4 ops) + 4 custom-stage assertions
      (3 functional + 1 playwright). Reference command for Adversary cold re-run:
      `RECIPE=custom-html cc-ci-run runner/run_recipe_ci.py`.
- **Per-recipe contract artifact landed**
  - `tests/custom-html/PARITY.md` — parity row for `health_check.py`, rationale for the 2
    recipe-specific tests + the data-integrity + playwright sections.
  - `tests/custom-html/functional/{test_health_check.py,test_content_roundtrip.py,test_content_type_header.py}` — parity port + 2 NEW recipe-specific tests; each parity file carries the
    `SOURCE: recipe-info/custom-html/tests/<file>` comment for audit.
  - `tests/custom-html/playwright/test_browser_smoke.py` — Phase-2 P6 home.

**Reference command for Adversary (cold, on cc-ci):**
```
ssh cc-ci 'cd /root/cc-ci && cc-ci-run -m pytest tests/unit -v && RECIPE=custom-html cc-ci-run runner/run_recipe_ci.py'
```

## Blocked
(none) — bootstrap access re-verified @2026-05-28: `ssh cc-ci` ok (root, NixOS 24.11), Gitea API
HTTP 200, wildcard DNS resolves to gateway 143.244.213.108.

## Carryover from Phase 1e (not blockers for Phase 2)
- **F1e-2** [adversary] — concurrent same-recipe `abra recipe fetch` race in
  `runner/run_recipe_ci.py::fetch_recipe`. Pre-existing in Phase 1d; not a 1e regression. Drone
  caps `MAX_TESTS=1` today, so practical impact bounded. Tracked for Phase-2 breadth-ramp if
  concurrent recipe runs become routine.