autonomic-bot
695a06aedd
Q1.1 custom-html: parity port + 2 NEW recipe-specific + playwright (Q0 PASS evidence stands). Q1.2 n8n: parity port + 2 NEW recipe-specific (rest_settings, login_state — both reject the 'n8n is starting up' placeholder, so non-vacuous). install overlay now polls page.goto until status==200 (absorbs n8n's /healthz-200-before-/-route-registered boot race). Q1.3 n8n backup data-integrity: covered by Phase-1d/1e lifecycle overlay pattern (volume marker survives backup→mutate→restore — PASSED in Q1.2 e2e). Q1.4 CLAIMED. Cold evidence: ssh cc-ci 'RECIPE=n8n cc-ci-run runner/run_recipe_ci.py' all 5 stages PASS, deploy-count=1, head_ref==chaos-version (HC1 non-vacuous), version moved 3.1.0+2.9.4 -> 3.2.0+2.20.6. Q1.2 note: deferred 'create workflow via API' from plan §4.3 in favor of /rest/settings + /rest/login JSON-shape assertions (equally non-vacuous, no owner-setup state to manage); recorded in BACKLOG-2 + JOURNAL-2. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
116 lines
8.5 KiB
Markdown
116 lines
8.5 KiB
Markdown
# BACKLOG — Phase 2 (per-recipe test authoring)
|
||
|
||
Phase-namespaced backlog. Builder edits `## Build backlog`; Adversary edits `## Adversary findings`.
|
||
Phase plan: `/srv/cc-ci/cc-ci-plan/plan-phase2-recipe-tests.md`
|
||
|
||
## Build backlog
|
||
|
||
### Q0 — Harness additions
|
||
- [x] **Q0.1** — `runner/harness/http.py` landed (canonical Phase-2 recipe-test HTTP API:
|
||
`http_get`/`http_post`/`http_request`/`retry_http_get`/`retry_http_post`/`wait_for_http`/
|
||
`assert_converges`). TTY abra wrapper already present (`runner/harness/abra.py::_run_pty`)
|
||
from Phase 1d. 11 unit tests landed.
|
||
- [x] **Q0.2** — `discovery.custom_tests` recurses into `tests/<recipe>/{functional,playwright}/`
|
||
(Phase 2 §4.1 layout); 2 unit tests landed.
|
||
- [x] **Q0.3** — `tests/custom-html/PARITY.md` landed (parity row for health_check + rationale for
|
||
2 new recipe-specific tests + data-integrity + playwright sections). Parity port:
|
||
`tests/custom-html/functional/test_health_check.py` (SOURCE comment present).
|
||
- [ ] **Q0.4** — Dependency resolver harness primitive (read `tests/<recipe>/recipe.toml`
|
||
`requires`/`test_requires`, deploy deps before the recipe under test, tear down with it). Mind
|
||
`MAX_TESTS`/node budget; sequence heavy ones. **Deferred to Q2** (needed once SSO providers come
|
||
online; no Phase-2 recipe in Q1 needs deps). Tracked in BACKLOG.
|
||
- [x] **Q0.5** — **RE-CLAIMED @2026-05-28** (commit `5741e88` adds F2-1 fix to original Q0).
|
||
Custom-html reference recipe runs the full parity + ≥2 specific + playwright suite green on
|
||
cc-ci; deploy-count=1; DECISIONS.md Phase-2 section in place. F2-1 closed by Builder; 21/21
|
||
unit tests PASS cold. Awaiting Adversary cold re-verify.
|
||
|
||
### Q1 — Pattern proof (custom-html + n8n)
|
||
- [x] **Q1.1** — custom-html: 2 NEW recipe-specific functional tests landed
|
||
(`test_content_roundtrip.py` + `test_content_type_header.py`); already cold-verified in Q0 PASS.
|
||
- [x] **Q1.2** — n8n enrolled under cc-ci (already had lifecycle overlays from Phase 1d/1e). Parity
|
||
port `tests/n8n/functional/test_health_check.py` + 2 NEW recipe-specific functional tests
|
||
(`test_rest_settings.py` + `test_login_state.py`) + PARITY.md complete. Install overlay's
|
||
Playwright now polls page.goto until status==200 (absorbs n8n boot variance). Note: the plan's
|
||
"(a) create a workflow via API, execute it" idea was deferred — n8n's REST API requires owner
|
||
setup before workflows are creatable, and the simpler /rest/settings + /rest/login JSON-shape
|
||
tests are equally non-vacuous (reject the "starting up" placeholder) and don't require
|
||
generating an owner password. Logged as a NOTE in PARITY.md; "≥2 specific" floor met.
|
||
- [x] **Q1.3** — n8n real backup data-integrity already covered by the Phase-1d/1e lifecycle overlay
|
||
pattern (`ops.pre_backup` seeds "original" in /home/node/.n8n; `pre_restore` mutates; restore
|
||
must return "original" — passed in the Q1.2 e2e run).
|
||
- [x] **Q1.4** — **CLAIMED @2026-05-28** (commit `2f3d5aa`). Both recipes green via the run path;
|
||
both PARITY.md complete. Awaiting Adversary cold-verify gate PASS.
|
||
|
||
### Q2 — SSO providers (keycloak + authentik)
|
||
- [ ] **Q2.1** — keycloak: port `tests/keycloak/oidc_integration.py` (the dependent-recipe test) and
|
||
`tests/health_check.py`. Add specific tests from plan §4.3 (realm+client via admin API; password
|
||
and client-credentials token grants; JWT claims).
|
||
- [ ] **Q2.2** — authentik: mirror the upstream repo if needed (per recipe mirror+PR flow); port
|
||
health_check + add specific tests.
|
||
- [ ] **Q2.3** — Reusable SSO-setup/OIDC-flow harness primitive: deploy provider → setup realm/client/
|
||
test-user (port `recipe-info/<dep>/setup_<provider>_integration.py`) → persist credentials
|
||
per-run → "full OIDC login → token → protected API call" assertion. Implement once in
|
||
`runner/harness/`; reused by every SSO-dependent recipe.
|
||
- [ ] **Q2.4** — Q2 gate: a dependent recipe deploys its provider + runs an OIDC login test in one run.
|
||
|
||
### Q3 — SSO-dependent suite (lasuite-docs, lasuite-drive, lasuite-meet, cryptpad, immich)
|
||
- [ ] **Q3.1** — lasuite-docs: parity (health_check, oidc_login, upload_conversion) + specific
|
||
(create-a-doc + WOPI discovery).
|
||
- [ ] **Q3.2** — lasuite-drive: enroll (mirror via recipe mirror+PR flow if absent); parity + specific
|
||
(upload to workspace, list/download; MinIO bucket present).
|
||
- [ ] **Q3.3** — lasuite-meet: parity (health_check, oidc_login, meeting_flow, webrtc-media,
|
||
webrtc-relay) + specific (create-a-room, two-user LiveKit token issuance, ICE-candidate gathering).
|
||
- [ ] **Q3.4** — cryptpad: parity (health_check, oidc_login) + specific (Playwright pad create+persist
|
||
— JS-rendered so curl insufficient).
|
||
- [ ] **Q3.5** — immich: enroll (mirror as needed); add specific (upload asset, list it back,
|
||
thumbnail/derivative).
|
||
- [ ] **Q3.6** — Q3 gate: each green with deps deployed, within node budget; SSO setup automated.
|
||
|
||
### Q4 — Remaining recipes
|
||
- [ ] **Q4.1** — matrix-synapse: parity (port shell tests as Python; `compress_state`,
|
||
`test_complexity_limit`, `test_purge`) + specific (register two users; one sends a message, the
|
||
other reads it; media upload→download; `/_matrix/federation/v1/version` reachable).
|
||
- [ ] **Q4.2** — mumble: enroll; specific (connect a client/CLI, channel presence beyond TCP health).
|
||
- [ ] **Q4.3** — bluesky-pds: parity (port `goat_account`) + specific (atproto post round-trip,
|
||
then delete account).
|
||
- [ ] **Q4.4** — ghost: enroll; specific (create-a-post round-trip).
|
||
- [ ] **Q4.5** — mattermost-lts: enroll; specific (create-a-message round-trip).
|
||
- [ ] **Q4.6** — discourse: enroll; specific (create-a-topic round-trip).
|
||
- [ ] **Q4.7** — plausible: enroll; specific (track a test event, query it back).
|
||
- [ ] **Q4.8** — uptime-kuma: enroll; specific (create a monitor, list it).
|
||
- [ ] **Q4.9** — mailu: enroll; specific (create a mailbox, send/receive verification).
|
||
- [ ] **Q4.10** — drone: enroll; specific (create/list builds via API).
|
||
- [ ] **Q4.11** — Q4 gate: each recipe green with parity + specific.
|
||
|
||
### Q5 — Completeness + docs
|
||
- [ ] **Q5.1** — `docs/enroll-recipe.md` updated with the per-recipe test contract (§4.1), the
|
||
`functional/` and `playwright/` subdirectory layout, the PARITY.md convention, the dependency
|
||
resolver hook, the SSO-setup harness — with a worked example.
|
||
- [ ] **Q5.2** — Adversary samples a subset and cold-verifies parity tables + specific tests are real
|
||
(not health-only, not skipped). NO weakened test, no corners cut (P7).
|
||
- [ ] **Q5.3** — Phase 2 `## DONE` after all P1–P8 Adversary cold-verified PASS, no standing VETO.
|
||
|
||
## Adversary findings
|
||
|
||
- [x] **F2-1 [adversary] — CLOSED @2026-05-28** by Builder commit `5741e88` (synthetic recipe +
|
||
monkeypatched `discovery.cc_ci_dir`, exactly the prescribed fix pattern from sibling
|
||
`test_discovery_phase2.py`). Adversary cold re-verify on `/root/adv-verify` @ HEAD `0b834e9`:
|
||
`cc-ci-run -m pytest tests/unit -v` → **21 passed in 4.69s** (the previously-failing
|
||
`test_custom_tests_repo_local_gated` now PASSes; no other regression). E2E PASS from prior
|
||
verdict at HEAD `d480411` still stands (only `tests/unit/test_discovery.py` + `tests/n8n/
|
||
PARITY.md` changed since; no harness/lifecycle code touched). Q0 PASS in REVIEW-2.
|
||
|
||
- [ ] **F2-2 [adversary] — scope/transparency observation, NOT a gate-blocker** — Phase-2 plan §6
|
||
Q0 lists 5 harness primitives ("HTTP/convergence, OIDC-flow, dependency resolver, backup
|
||
data-integrity, TTY abra"). Q0 changeset ships HTTP/convergence (`runner/harness/http.py`) +
|
||
TTY abra (reused from `runner/harness/abra.py::_run_pty`, Phase 1d). OIDC-flow + dependency
|
||
resolver + a dedicated backup-data-integrity primitive are NOT in the changeset. BACKLOG-2
|
||
`Q0.4` (Dependency resolver) is still `[ ]` open; BACKLOG-2 `Q0.1` mentions "Backup data-
|
||
integrity primitive" but the implementation reuses Phase-1e `lifecycle.exec_in_app`
|
||
directly. This is consistent with deferring primitives until their consuming recipe (Q2
|
||
keycloak/authentik for OIDC; Q3 dependent recipes for dep resolver) needs them, and with
|
||
Q0's narrower acceptance ("custom-html — which has no SSO/deps — uses them"). NOT a Q0
|
||
gate-blocker, but Q0 cannot be considered "complete" in the broad sense of the §6 enumeration
|
||
until those primitives ship in Q2/Q3. Recording so a future Q2/Q3 verdict checks them off.
|
||
- Filed by Adversary @2026-05-28.
|