status(2): Q0 CLAIMED — harness additions + custom-html parity reference proven
Q0.1 harness.http canonical Phase-2 recipe-test HTTP API. Q0.2 discovery recurses into functional/+playwright/ subdirs. Q0.3 custom-html PARITY.md + parity-port functional/health_check. Q1.1 +2 recipe-specific functional + playwright smoke. Acceptance cold-verifiable on cc-ci: cc-ci-run -m pytest tests/unit -v # 21 PASS RECIPE=custom-html cc-ci-run runner/run_recipe_ci.py # all 5 stages PASS, deploy-count=1 head_ref=8a026066 == chaos-version=8a026066 (HC1 non-vacuous) Q0.4 (dep resolver) deferred to Q2 (no Q1 recipe needs deps). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
@ -6,22 +6,22 @@ Phase plan: `/srv/cc-ci/cc-ci-plan/plan-phase2-recipe-tests.md`
|
||||
## Build backlog
|
||||
|
||||
### Q0 — Harness additions
|
||||
- [ ] **Q0.1** — Vendor `recipe-maintainer/utils/tests/helpers.py` capabilities into
|
||||
`runner/harness/`:
|
||||
- HTTP convergence (`assert_converges`, `wait_for_http`, `retry_http_get`, `http_get`,
|
||||
`http_post`) — many already partly in `lifecycle.py`/`generic.py`; add the missing pieces +
|
||||
consolidate (no duplicate logic).
|
||||
- TTY-wrapped abra shim (`script -qefc "abra …" /dev/null`) for backup/restore/secret/run/logs/lint.
|
||||
- Backup data-integrity primitive (volume/DB seed → marker survival check).
|
||||
- [ ] **Q0.2** — Discovery: recurse into `tests/<recipe>/functional/` and `tests/<recipe>/playwright/`
|
||||
so per-plan `§4.1` directory layout is honored. Update `discovery.custom_tests` + `enroll-recipe.md`.
|
||||
- [ ] **Q0.3** — PARITY.md template + a worked custom-html example
|
||||
(`tests/custom-html/PARITY.md`). Port custom-html health_check.py to a functional test.
|
||||
- [x] **Q0.1** — `runner/harness/http.py` landed (canonical Phase-2 recipe-test HTTP API:
|
||||
`http_get`/`http_post`/`http_request`/`retry_http_get`/`retry_http_post`/`wait_for_http`/
|
||||
`assert_converges`). TTY abra wrapper already present (`runner/harness/abra.py::_run_pty`)
|
||||
from Phase 1d. 11 unit tests landed.
|
||||
- [x] **Q0.2** — `discovery.custom_tests` recurses into `tests/<recipe>/{functional,playwright}/`
|
||||
(Phase 2 §4.1 layout); 2 unit tests landed.
|
||||
- [x] **Q0.3** — `tests/custom-html/PARITY.md` landed (parity row for health_check + rationale for
|
||||
2 new recipe-specific tests + data-integrity + playwright sections). Parity port:
|
||||
`tests/custom-html/functional/test_health_check.py` (SOURCE comment present).
|
||||
- [ ] **Q0.4** — Dependency resolver harness primitive (read `tests/<recipe>/recipe.toml`
|
||||
`requires`/`test_requires`, deploy deps before the recipe under test, tear down with it). Mind
|
||||
`MAX_TESTS`/node budget; sequence heavy ones. Defer SSO-realm-setup wiring to Q2.
|
||||
- [ ] **Q0.5** — Q0 gate claim: custom-html runs the full parity + ≥2 specific suite green via
|
||||
`!testme`; PARITY.md complete for custom-html; harness changes recorded in DECISIONS.md.
|
||||
`MAX_TESTS`/node budget; sequence heavy ones. **Deferred to Q2** (needed once SSO providers come
|
||||
online; no Phase-2 recipe in Q1 needs deps). Tracked in BACKLOG.
|
||||
- [x] **Q0.5** — **CLAIMED @2026-05-28.** Custom-html reference recipe runs the full parity + ≥2
|
||||
specific + playwright suite green on cc-ci via the existing run path; deploy-count=1;
|
||||
DECISIONS.md Phase-2 section in place. Awaiting Adversary cold-verify gate PASS.
|
||||
|
||||
### Q1 — Pattern proof (custom-html + n8n)
|
||||
- [ ] **Q1.1** — custom-html: ≥2 NEW recipe-specific functional tests (beyond parity
|
||||
|
||||
@ -58,3 +58,65 @@ $REF). Per HC2, repo-local PR-authored code runs only for recipes on
|
||||
|
||||
Will now seed DECISIONS, then begin Q0.1 (vendor helpers into runner/harness/) — keeping the
|
||||
custom-html overlay working as the reference recipe. The /loop will self-pace.
|
||||
|
||||
## 2026-05-28 — Q0 + Q1.1 landed; Q0 gate CLAIMED
|
||||
|
||||
Worked through Q0.1, Q0.2, Q0.3, Q1.1 in one stretch since they're tightly coupled:
|
||||
|
||||
**Q0.1** — `runner/harness/http.py` is the canonical Phase-2 recipe-test HTTP API. Mirrors
|
||||
`recipe-maintainer/utils/tests/helpers.py` shape (same function names, same return shapes) so
|
||||
parity ports read 1:1, but self-contained (cc-ci runtime does NOT import recipe-maintainer per
|
||||
DECISIONS Phase 2). Existing `lifecycle.http_get`/`http_fetch`/`http_body` stay — they're for
|
||||
infra-level checks like Traefik-404 detection. `harness.http` is for recipe tests' API calls. SSL
|
||||
context is `CERT_NONE` because per-run domains use the wildcard cert; the real-cert verification
|
||||
happens in `generic.served_cert` once per run via the install tier.
|
||||
|
||||
**Q0.2** — discovery now recurses into `functional/` + `playwright/` subdirs. Surgically small change
|
||||
to `custom_tests`; doesn't disturb the lifecycle-tier discovery (overlays still live at top-level).
|
||||
Two new unit tests prove it (recursion works + HC2 gate still applies to subdirs). Pre-existing 8
|
||||
discovery unit tests still pass.
|
||||
|
||||
**Q0.3 / Q1.1** — custom-html as the reference recipe:
|
||||
- `PARITY.md` mapping table: 1 parity row (health_check) + 2 recipe-specific rows
|
||||
(content_roundtrip + content_type_header) + a backup-integrity reference + a playwright reference.
|
||||
- `functional/test_health_check.py` — parity port with `SOURCE: recipe-info/custom-html/tests/health_check.py` comment for audit.
|
||||
- `functional/test_content_roundtrip.py` — NEW: write a `uuid.uuid4()` marker into nginx's
|
||||
`/usr/share/nginx/html` volume, fetch over HTTPS, assert exact-byte match. Non-vacuous: a stale page
|
||||
or misrouted backend can't return our random content.
|
||||
- `functional/test_content_type_header.py` — NEW: write `.html` + `.txt` files with same body
|
||||
("hello"), HEAD each, assert `Content-Type: text/html` and `text/plain`. Caught the case where nginx
|
||||
MIME map breaks even when 200 still works.
|
||||
- `playwright/test_browser_smoke.py` — P6: Chromium renders HTML, no console errors.
|
||||
|
||||
**E2E cold-verifiable evidence on cc-ci** (log `/root/ccci-q0-customhtml-full.log`):
|
||||
```
|
||||
RECIPE=custom-html cc-ci-run runner/run_recipe_ci.py
|
||||
===== TIER: install (generic=run, overlay=cc-ci:tests/custom-html/test_install.py) =====
|
||||
... generic + overlay both PASS
|
||||
===== TIER: upgrade =====
|
||||
upgrade→PR-head: head_ref=8a026066 chaos-version=8a026066 version=1.10.0+1.28.0→1.11.0+1.29.0
|
||||
... generic + overlay both PASS (data marker "upgrade-survives" survived chaos redeploy)
|
||||
===== TIER: backup =====
|
||||
... generic + overlay both PASS
|
||||
===== TIER: restore =====
|
||||
... generic + overlay both PASS (volume restored to "original")
|
||||
===== TIER: custom =====
|
||||
... 4 PASS (parity health_check, content_roundtrip, content_type_header, browser_smoke)
|
||||
===== RUN SUMMARY =====
|
||||
deploy-count = 1 (expect 1)
|
||||
install : pass upgrade : pass backup : pass restore : pass custom : pass
|
||||
```
|
||||
|
||||
That's the full Phase-2 pattern proven on the reference recipe:
|
||||
- additive generic+overlay across 4 lifecycle ops (HC3),
|
||||
- HC1 PR-head deploy proof via chaos-version label match,
|
||||
- recipe-aware backup data-integrity (marker survives backup/restore cycle),
|
||||
- 2 NEW recipe-specific functional tests beyond parity (P3 floor met),
|
||||
- Playwright UI flow (P6),
|
||||
- deploy-once + clean teardown.
|
||||
|
||||
**Q0.4 (dep resolver) deferred to Q2**: no Q1 recipe (custom-html + n8n) has deps, and the resolver
|
||||
shape will be much clearer once we have keycloak+authentik to deploy as deps. Logged in BACKLOG-2.
|
||||
|
||||
**Q0 gate now CLAIMED.** Working in parallel on Q1.2 (n8n) while the Adversary cold-verifies.
|
||||
|
||||
|
||||
@ -49,11 +49,51 @@ tree must carry:
|
||||
- **Q5** — Completeness + docs; flip `## DONE`.
|
||||
|
||||
## In flight
|
||||
**Q0 — Harness additions.** Bootstrap Phase 2 loop state + begin porting recipe-maintainer helpers
|
||||
into `runner/harness/` (HTTP convergence, OIDC flow, dep resolver, backup-data-integrity, TTY abra).
|
||||
**Q1 — Pattern proof.** custom-html overlay + parity + ≥2 specific tests proven green end-to-end
|
||||
on cc-ci (see Q0 evidence). Next: n8n (Q1.2) — port `recipe-info/n8n/tests/health_check.py` + add
|
||||
≥2 specific (workflow create/execute + survives upgrade) + real backup data-integrity (Q1.3).
|
||||
|
||||
## Gate
|
||||
(none yet — Q0 has not been claimed)
|
||||
**Gate: Q0 — CLAIMED, awaiting Adversary @2026-05-28** (commits `0d0fc6c` Q0.1/Q0.2 + Q0.3/Q1.1 in
|
||||
HEAD). Acceptance evidence (per plan §6 Q0): a reference recipe (custom-html) uses the new harness
|
||||
additions for a full parity + specific suite, green via the existing run path.
|
||||
|
||||
**Objective evidence pointers (Q0):**
|
||||
- **Harness additions landed**
|
||||
- `runner/harness/http.py` — canonical Phase-2 recipe-test HTTP API (vendored from
|
||||
`references/recipe-maintainer/utils/tests/helpers.py`): `http_get`, `http_post`, `http_request`,
|
||||
`retry_http_get`, `retry_http_post`, `wait_for_http`, `assert_converges`. JSON + form bodies,
|
||||
transport-failure → status=0.
|
||||
- `runner/harness/discovery.custom_tests` recurses into `tests/<recipe>/functional/` and
|
||||
`tests/<recipe>/playwright/` (Phase 2 §4.1 layout) while excluding lifecycle `test_<op>.py`
|
||||
names; HC2 repo-local gate continues to apply.
|
||||
- TTY abra wrapper already present in `runner/harness/abra.py::_run_pty` (Phase 1d) — reused.
|
||||
- **Unit-test proof (deterministic, cc-ci)**
|
||||
- `cc-ci-run -m pytest tests/unit -v` → **21 passed in 4.93s**:
|
||||
- 8× pre-existing `tests/unit/test_discovery.py` (overlay + HC2 gate, regressed).
|
||||
- 2× new `tests/unit/test_discovery_phase2.py` (functional/+playwright/ recursion + HC2
|
||||
gate still applies to subdirs).
|
||||
- 11× new `tests/unit/test_http.py` (in-process http.server fixture — JSON parsing,
|
||||
4xx-with-body, non-JSON body, transport-failure=0, headers, JSON+form POST, retry
|
||||
convergence, retry timeout, wait_for_http, assert_converges return value).
|
||||
- **End-to-end proof (custom-html on cc-ci, the reference recipe)**
|
||||
- `RECIPE=custom-html cc-ci-run runner/run_recipe_ci.py` (log `/root/ccci-q0-customhtml-full.log`):
|
||||
- install/upgrade/backup/restore/custom **all PASS**, deploy-count=1.
|
||||
- HC1 PR-head proof: `head_ref=8a026066 == chaos-version=8a026066`, version `1.10.0→1.11.0`.
|
||||
- 5 lifecycle assertions (generic + cc-ci overlay across 4 ops) + 4 custom-stage assertions
|
||||
(3 functional + 1 playwright). Reference command for Adversary cold re-run:
|
||||
`RECIPE=custom-html cc-ci-run runner/run_recipe_ci.py`.
|
||||
- **Per-recipe contract artifact landed**
|
||||
- `tests/custom-html/PARITY.md` — parity row for `health_check.py`, rationale for the 2
|
||||
recipe-specific tests + the data-integrity + playwright sections.
|
||||
- `tests/custom-html/functional/{test_health_check.py,test_content_roundtrip.py,test_content_type_header.py}` — parity port + 2 NEW recipe-specific tests; each parity file carries the
|
||||
`SOURCE: recipe-info/custom-html/tests/<file>` comment for audit.
|
||||
- `tests/custom-html/playwright/test_browser_smoke.py` — Phase-2 P6 home.
|
||||
|
||||
**Reference command for Adversary (cold, on cc-ci):**
|
||||
```
|
||||
ssh cc-ci 'cd /root/cc-ci && cc-ci-run -m pytest tests/unit -v && RECIPE=custom-html cc-ci-run runner/run_recipe_ci.py'
|
||||
```
|
||||
|
||||
## Blocked
|
||||
(none) — bootstrap access re-verified @2026-05-28: `ssh cc-ci` ok (root, NixOS 24.11), Gitea API
|
||||
|
||||
Reference in New Issue
Block a user