status(2): Q0 RE-CLAIMED — F2-1 fix verified cold (21/21 unit PASS)
Per Adversary cold (REVIEW-2 "Q0 FAIL"), F2-1 mechanical regression: the Phase-1e HC2 unit test
asserted custom_tests('custom-html', rl) == [] when the real custom-html dir had no functional/
tests. Phase-2 added 4 legit functional/playwright files there, so the assertion no longer holds.
Behavior is correct; the test fixture was brittle.
Fix landed commit 5741e88: switch the assertion to a synthetic recipe + monkeypatch cc_ci_dir
(same pattern as the Phase-2 sibling test_discovery_phase2.py). Cold re-run: 21/21 PASS.
F2-2 (Q0 scope observation): OIDC-flow + dep resolver primitives deferred to Q2/Q3 when consuming
recipes land; BACKLOG-2 Q0.4 explicitly tracks this — acknowledged in STATUS-2 gate text.
Q0 RE-CLAIMED, awaiting Adversary re-verify.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
@ -19,9 +19,10 @@ Phase plan: `/srv/cc-ci/cc-ci-plan/plan-phase2-recipe-tests.md`
|
||||
`requires`/`test_requires`, deploy deps before the recipe under test, tear down with it). Mind
|
||||
`MAX_TESTS`/node budget; sequence heavy ones. **Deferred to Q2** (needed once SSO providers come
|
||||
online; no Phase-2 recipe in Q1 needs deps). Tracked in BACKLOG.
|
||||
- [x] **Q0.5** — **CLAIMED @2026-05-28.** Custom-html reference recipe runs the full parity + ≥2
|
||||
specific + playwright suite green on cc-ci via the existing run path; deploy-count=1;
|
||||
DECISIONS.md Phase-2 section in place. Awaiting Adversary cold-verify gate PASS.
|
||||
- [x] **Q0.5** — **RE-CLAIMED @2026-05-28** (commit `5741e88` adds F2-1 fix to original Q0).
|
||||
Custom-html reference recipe runs the full parity + ≥2 specific + playwright suite green on
|
||||
cc-ci; deploy-count=1; DECISIONS.md Phase-2 section in place. F2-1 closed by Builder; 21/21
|
||||
unit tests PASS cold. Awaiting Adversary cold re-verify.
|
||||
|
||||
### Q1 — Pattern proof (custom-html + n8n)
|
||||
- [ ] **Q1.1** — custom-html: ≥2 NEW recipe-specific functional tests (beyond parity
|
||||
|
||||
@ -120,3 +120,35 @@ shape will be much clearer once we have keycloak+authentik to deploy as deps. Lo
|
||||
|
||||
**Q0 gate now CLAIMED.** Working in parallel on Q1.2 (n8n) while the Adversary cold-verifies.
|
||||
|
||||
|
||||
## 2026-05-28 — F2-1 fix: synthetic-recipe fixture (Adversary FAIL on Q0)
|
||||
|
||||
The Adversary FAILed Q0 cold on F2-1: `tests/unit/test_discovery.py::test_custom_tests_repo_local_gated` (Phase-1e HC2 test) used the real recipe name `"custom-html"` and asserted
|
||||
`custom_tests("custom-html", repo_local) == []`. Phase-2 commit `bec9265` added 4 legit non-lifecycle
|
||||
tests under `tests/custom-html/{functional,playwright}/`, which `custom_tests()` now correctly
|
||||
returns — so the `== []` assertion no longer holds. Behavior is right; the fixture was brittle.
|
||||
|
||||
My "21 passed" evidence was real on the Builder clone — but I had synced the new tests to cc-ci
|
||||
**before** syncing the new custom-html functional/ tests, so at that moment the assertion still held.
|
||||
The Adversary's cold re-run from origin/main pulled the full state and correctly caught the regression.
|
||||
|
||||
**Fix (commit `5741e88`):** switch to synthetic recipe + monkeypatch `discovery.cc_ci_dir` — same
|
||||
pattern already used in the Phase-2 sibling `tests/unit/test_discovery_phase2.py`. 5-line change,
|
||||
no behavior change. Cold-verifiable: `cc-ci-run -m pytest tests/unit -v` → 21/21 PASS.
|
||||
|
||||
F2-2 (scope observation) — the Adversary flagged that Q0.4 (dep resolver) and OIDC-flow primitive
|
||||
are not yet implemented; explicitly deferred to Q2/Q3 in BACKLOG-2. Acknowledged in STATUS-2 gate
|
||||
text.
|
||||
|
||||
**Lesson:** when adding new content to an existing recipe directory, scan the unit tests for any
|
||||
that assume that directory is empty/lifecycle-only. The synthetic-recipe + monkeypatch pattern is
|
||||
the right shape for all such unit tests; we should prefer it across the board.
|
||||
|
||||
**n8n probe ran in the background to validate endpoint shapes for Q1.2:**
|
||||
- `/` → 200 text/html (the SPA)
|
||||
- `/healthz` → 200 `{"status":"ok"}` (already used by install overlay)
|
||||
- `/types/nodes.json` → 200 but size=31 bytes, not JSON (probably SPA fallback). REJECT this idea.
|
||||
- Probe terminated before reaching `/rest/settings` / `/rest/login` (the JSON parse on
|
||||
`/types/nodes.json` raised). Re-running probe now without the JSON gate.
|
||||
|
||||
Q0 re-claimed; awaiting Adversary re-verify. Continuing on Q1.2 (n8n) in parallel.
|
||||
|
||||
@ -54,9 +54,13 @@ on cc-ci (see Q0 evidence). Next: n8n (Q1.2) — port `recipe-info/n8n/tests/hea
|
||||
≥2 specific (workflow create/execute + survives upgrade) + real backup data-integrity (Q1.3).
|
||||
|
||||
## Gate
|
||||
**Gate: Q0 — CLAIMED, awaiting Adversary @2026-05-28** (commits `0d0fc6c` Q0.1/Q0.2 + Q0.3/Q1.1 in
|
||||
HEAD). Acceptance evidence (per plan §6 Q0): a reference recipe (custom-html) uses the new harness
|
||||
additions for a full parity + specific suite, green via the existing run path.
|
||||
**Gate: Q0 — RE-CLAIMED, awaiting Adversary @2026-05-28** (commit `5741e88` — F2-1 fix landed on
|
||||
top of the original Q0 changeset). Acceptance evidence (per plan §6 Q0): a reference recipe
|
||||
(custom-html) uses the new harness additions for a full parity + specific suite, green via the
|
||||
existing run path. F2-1 (test_custom_tests_repo_local_gated stale assertion) closed by Builder; cold
|
||||
re-run on cc-ci → **21/21 PASS** including the previously-failing test. F2-2 (scope observation:
|
||||
OIDC-flow + dep resolver not in Q0) acknowledged — those primitives implement when Q2/Q3 consume
|
||||
them; BACKLOG-2 Q0.4 remains open and explicitly deferred.
|
||||
|
||||
**Objective evidence pointers (Q0):**
|
||||
- **Harness additions landed**
|
||||
@ -68,8 +72,9 @@ additions for a full parity + specific suite, green via the existing run path.
|
||||
`tests/<recipe>/playwright/` (Phase 2 §4.1 layout) while excluding lifecycle `test_<op>.py`
|
||||
names; HC2 repo-local gate continues to apply.
|
||||
- TTY abra wrapper already present in `runner/harness/abra.py::_run_pty` (Phase 1d) — reused.
|
||||
- **Unit-test proof (deterministic, cc-ci)**
|
||||
- `cc-ci-run -m pytest tests/unit -v` → **21 passed in 4.93s**:
|
||||
- **Unit-test proof (deterministic, cc-ci; post-F2-1 fix commit `5741e88`)**
|
||||
- `cc-ci-run -m pytest tests/unit -v` → **21 passed in 5.38s** (the previously-failing
|
||||
`test_custom_tests_repo_local_gated` now passes; synthetic-recipe + monkeypatch fixture):
|
||||
- 8× pre-existing `tests/unit/test_discovery.py` (overlay + HC2 gate, regressed).
|
||||
- 2× new `tests/unit/test_discovery_phase2.py` (functional/+playwright/ recursion + HC2
|
||||
gate still applies to subdirs).
|
||||
|
||||
Reference in New Issue
Block a user