autonomic-bot
0b834e90f2
Per Adversary cold (REVIEW-2 "Q0 FAIL"), F2-1 mechanical regression: the Phase-1e HC2 unit test
asserted custom_tests('custom-html', rl) == [] when the real custom-html dir had no functional/
tests. Phase-2 added 4 legit functional/playwright files there, so the assertion no longer holds.
Behavior is correct; the test fixture was brittle.
Fix landed commit 5741e88: switch the assertion to a synthetic recipe + monkeypatch cc_ci_dir
(same pattern as the Phase-2 sibling test_discovery_phase2.py). Cold re-run: 21/21 PASS.
F2-2 (Q0 scope observation): OIDC-flow + dep resolver primitives deferred to Q2/Q3 when consuming
recipes land; BACKLOG-2 Q0.4 explicitly tracks this — acknowledged in STATUS-2 gate text.
Q0 RE-CLAIMED, awaiting Adversary re-verify.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
155 lines
9.4 KiB
Markdown
155 lines
9.4 KiB
Markdown
# JOURNAL — Phase 2 (per-recipe test authoring)
|
||
|
||
Builder-private (append-only). Builder rationalisations, dead-ends, in-the-moment reasoning. The
|
||
Adversary does NOT read this before forming a verdict; objective evidence goes in STATUS-2 / REVIEW-2.
|
||
Phase plan: `/srv/cc-ci/cc-ci-plan/plan-phase2-recipe-tests.md`
|
||
|
||
---
|
||
|
||
## 2026-05-28 — Phase 2 bootstrap
|
||
|
||
Phase 1e completed @2026-05-28 (commit 0fe1218, NO VETO, all HC1–HC4 Adversary cold-verified PASS).
|
||
Foundation is in place: the orchestrator deploys ONCE per run, performs each lifecycle op ONCE
|
||
(install→deploy / upgrade→chaos-redeploy of PR head / backup→`abra app backup` / restore→`abra app
|
||
restore`), and runs **both** generic (`tests/_generic/test_<op>.py`) and overlay
|
||
(`tests/<recipe>/test_<op>.py`) assertion files **additively** against the shared post-op state.
|
||
Pre-op seeds live in optional `tests/<recipe>/ops.py` (`pre_install`/`pre_upgrade`/`pre_backup`/
|
||
`pre_restore`). The deploy-count guard (DG4.1) stays =1; teardown is sacred. Per Phase-1e HC1, the
|
||
upgrade tier proves PR-head was deployed via `chaos-version` label = `head_ref` (head SHA from
|
||
$REF). Per HC2, repo-local PR-authored code runs only for recipes on
|
||
`tests/repo-local-approved.txt` (default-deny).
|
||
|
||
**Bootstrap (this session):**
|
||
1. `git pull --rebase` — already up to date.
|
||
2. Verified §1 access: `ssh cc-ci` OK (NixOS 24.11), Gitea API HTTP 200, wildcard
|
||
`probe-$RANDOM.ci.commoninternet.net` resolves to gateway `143.244.213.108`.
|
||
3. Read the Phase-2 plan + plan.md §6.1/§7/§9 (loop protocol, single-writer ownership, gate
|
||
handshake, anti-drift). Read STATUS-1e + REVIEW-1e final to inherit the harness invariants
|
||
(HC1–HC4 cold-verified PASS, F1e-2 not blocking).
|
||
4. Surveyed existing state: `tests/<recipe>/` already exists for **custom-html, cryptpad, keycloak,
|
||
lasuite-docs, matrix-synapse, n8n** — these were built out as Phase-1d/1e overlays + recipe_meta
|
||
+ ops.py. The lifecycle overlay model (test_install/upgrade/backup/restore.py + ops.py) is the
|
||
foundation. Phase 2 adds **parity-port functional tests** + **≥2 NEW recipe-specific tests** +
|
||
**dependency/SSO resolver** + **PARITY.md** per recipe.
|
||
5. Surveyed `references/recipe-maintainer` (mounted at `/srv/recipe-maintainer/`) — the parity
|
||
source. Per-recipe corpus:
|
||
- **custom-html** — health_check.py (200 check)
|
||
- **n8n** — health_check.py
|
||
- **keycloak** — health_check.py + oidc_integration.py (cross-recipe with lasuite-docs)
|
||
- **cryptpad** — health_check.py + oidc_login.py
|
||
- **lasuite-docs** — health_check.py + oidc_login.py + upload_conversion.py
|
||
- **lasuite-meet** — health_check.py + oidc_login.py + meeting_flow.py + webrtc-media.py +
|
||
webrtc-relay.py
|
||
- **matrix-synapse** — *shell* tests: compress_state.sh + test_complexity_limit.sh + test_purge.sh
|
||
(will port semantics to Python under cc-ci)
|
||
- **hedgedoc / authentik / immich / bluesky-pds / mumble / gitea / lichen / lichen-markdown** —
|
||
no `tests/` dir under recipe-info yet, will fill from plan §4.3 spec.
|
||
|
||
**Plan-shape orientation:**
|
||
- `tests/<recipe>/test_<op>.py` (lifecycle overlays) — already established.
|
||
- `tests/<recipe>/functional/` — Phase-2 introduces this subdir for parity-port + new specific tests.
|
||
Discovery currently globs `test_*.py` at the top level only; will need to recurse (Q0.2).
|
||
- `tests/<recipe>/playwright/` — same.
|
||
- `tests/<recipe>/PARITY.md` — Phase-2 introduces this; mapping table per recipe.
|
||
|
||
**Bootstrap commits incoming:**
|
||
- Add STATUS-2.md / BACKLOG-2.md / JOURNAL-2.md (this session).
|
||
- DECISIONS.md append: PARITY.md format, functional/ + playwright/ subdirs, dep-resolver shape.
|
||
|
||
Will now seed DECISIONS, then begin Q0.1 (vendor helpers into runner/harness/) — keeping the
|
||
custom-html overlay working as the reference recipe. The /loop will self-pace.
|
||
|
||
## 2026-05-28 — Q0 + Q1.1 landed; Q0 gate CLAIMED
|
||
|
||
Worked through Q0.1, Q0.2, Q0.3, Q1.1 in one stretch since they're tightly coupled:
|
||
|
||
**Q0.1** — `runner/harness/http.py` is the canonical Phase-2 recipe-test HTTP API. Mirrors
|
||
`recipe-maintainer/utils/tests/helpers.py` shape (same function names, same return shapes) so
|
||
parity ports read 1:1, but self-contained (cc-ci runtime does NOT import recipe-maintainer per
|
||
DECISIONS Phase 2). Existing `lifecycle.http_get`/`http_fetch`/`http_body` stay — they're for
|
||
infra-level checks like Traefik-404 detection. `harness.http` is for recipe tests' API calls. SSL
|
||
context is `CERT_NONE` because per-run domains use the wildcard cert; the real-cert verification
|
||
happens in `generic.served_cert` once per run via the install tier.
|
||
|
||
**Q0.2** — discovery now recurses into `functional/` + `playwright/` subdirs. Surgically small change
|
||
to `custom_tests`; doesn't disturb the lifecycle-tier discovery (overlays still live at top-level).
|
||
Two new unit tests prove it (recursion works + HC2 gate still applies to subdirs). Pre-existing 8
|
||
discovery unit tests still pass.
|
||
|
||
**Q0.3 / Q1.1** — custom-html as the reference recipe:
|
||
- `PARITY.md` mapping table: 1 parity row (health_check) + 2 recipe-specific rows
|
||
(content_roundtrip + content_type_header) + a backup-integrity reference + a playwright reference.
|
||
- `functional/test_health_check.py` — parity port with `SOURCE: recipe-info/custom-html/tests/health_check.py` comment for audit.
|
||
- `functional/test_content_roundtrip.py` — NEW: write a `uuid.uuid4()` marker into nginx's
|
||
`/usr/share/nginx/html` volume, fetch over HTTPS, assert exact-byte match. Non-vacuous: a stale page
|
||
or misrouted backend can't return our random content.
|
||
- `functional/test_content_type_header.py` — NEW: write `.html` + `.txt` files with same body
|
||
("hello"), HEAD each, assert `Content-Type: text/html` and `text/plain`. Caught the case where nginx
|
||
MIME map breaks even when 200 still works.
|
||
- `playwright/test_browser_smoke.py` — P6: Chromium renders HTML, no console errors.
|
||
|
||
**E2E cold-verifiable evidence on cc-ci** (log `/root/ccci-q0-customhtml-full.log`):
|
||
```
|
||
RECIPE=custom-html cc-ci-run runner/run_recipe_ci.py
|
||
===== TIER: install (generic=run, overlay=cc-ci:tests/custom-html/test_install.py) =====
|
||
... generic + overlay both PASS
|
||
===== TIER: upgrade =====
|
||
upgrade→PR-head: head_ref=8a026066 chaos-version=8a026066 version=1.10.0+1.28.0→1.11.0+1.29.0
|
||
... generic + overlay both PASS (data marker "upgrade-survives" survived chaos redeploy)
|
||
===== TIER: backup =====
|
||
... generic + overlay both PASS
|
||
===== TIER: restore =====
|
||
... generic + overlay both PASS (volume restored to "original")
|
||
===== TIER: custom =====
|
||
... 4 PASS (parity health_check, content_roundtrip, content_type_header, browser_smoke)
|
||
===== RUN SUMMARY =====
|
||
deploy-count = 1 (expect 1)
|
||
install : pass upgrade : pass backup : pass restore : pass custom : pass
|
||
```
|
||
|
||
That's the full Phase-2 pattern proven on the reference recipe:
|
||
- additive generic+overlay across 4 lifecycle ops (HC3),
|
||
- HC1 PR-head deploy proof via chaos-version label match,
|
||
- recipe-aware backup data-integrity (marker survives backup/restore cycle),
|
||
- 2 NEW recipe-specific functional tests beyond parity (P3 floor met),
|
||
- Playwright UI flow (P6),
|
||
- deploy-once + clean teardown.
|
||
|
||
**Q0.4 (dep resolver) deferred to Q2**: no Q1 recipe (custom-html + n8n) has deps, and the resolver
|
||
shape will be much clearer once we have keycloak+authentik to deploy as deps. Logged in BACKLOG-2.
|
||
|
||
**Q0 gate now CLAIMED.** Working in parallel on Q1.2 (n8n) while the Adversary cold-verifies.
|
||
|
||
|
||
## 2026-05-28 — F2-1 fix: synthetic-recipe fixture (Adversary FAIL on Q0)
|
||
|
||
The Adversary FAILed Q0 cold on F2-1: `tests/unit/test_discovery.py::test_custom_tests_repo_local_gated` (Phase-1e HC2 test) used the real recipe name `"custom-html"` and asserted
|
||
`custom_tests("custom-html", repo_local) == []`. Phase-2 commit `bec9265` added 4 legit non-lifecycle
|
||
tests under `tests/custom-html/{functional,playwright}/`, which `custom_tests()` now correctly
|
||
returns — so the `== []` assertion no longer holds. Behavior is right; the fixture was brittle.
|
||
|
||
My "21 passed" evidence was real on the Builder clone — but I had synced the new tests to cc-ci
|
||
**before** syncing the new custom-html functional/ tests, so at that moment the assertion still held.
|
||
The Adversary's cold re-run from origin/main pulled the full state and correctly caught the regression.
|
||
|
||
**Fix (commit `5741e88`):** switch to synthetic recipe + monkeypatch `discovery.cc_ci_dir` — same
|
||
pattern already used in the Phase-2 sibling `tests/unit/test_discovery_phase2.py`. 5-line change,
|
||
no behavior change. Cold-verifiable: `cc-ci-run -m pytest tests/unit -v` → 21/21 PASS.
|
||
|
||
F2-2 (scope observation) — the Adversary flagged that Q0.4 (dep resolver) and OIDC-flow primitive
|
||
are not yet implemented; explicitly deferred to Q2/Q3 in BACKLOG-2. Acknowledged in STATUS-2 gate
|
||
text.
|
||
|
||
**Lesson:** when adding new content to an existing recipe directory, scan the unit tests for any
|
||
that assume that directory is empty/lifecycle-only. The synthetic-recipe + monkeypatch pattern is
|
||
the right shape for all such unit tests; we should prefer it across the board.
|
||
|
||
**n8n probe ran in the background to validate endpoint shapes for Q1.2:**
|
||
- `/` → 200 text/html (the SPA)
|
||
- `/healthz` → 200 `{"status":"ok"}` (already used by install overlay)
|
||
- `/types/nodes.json` → 200 but size=31 bytes, not JSON (probably SPA fallback). REJECT this idea.
|
||
- Probe terminated before reaching `/rest/settings` / `/rest/login` (the JSON parse on
|
||
`/types/nodes.json` raised). Re-running probe now without the JSON gate.
|
||
|
||
Q0 re-claimed; awaiting Adversary re-verify. Continuing on Q1.2 (n8n) in parallel.
|