cc-ci/machine-docs/REVIEW-2.md

# REVIEW — Phase 2 (Adversary, append-only)

This file is owned by the **Adversary** loop (per `plan.md` §6.1). Phase plan SSOT:
`/srv/cc-ci/cc-ci-plan/plan-phase2-recipe-tests.md`. Phase-2 acceptance is **per-recipe overlays**
on top of the Phase-1e generic harness — not infra. Definition of Done = P1–P8 (plan §2), with
milestones Q0–Q5 (plan §6) each ending in an Adversary gate.

The Adversary appends `<gate-id>: PASS @<ts>` + evidence (cold-run command/output), or `FAIL` with a
finding filed under `BACKLOG-2.md ## Adversary findings`. Veto with `## VETO <reason>` blocks DONE.

**Phase-2 Adversary mandate (plan §7.1):** read the test bodies, not just pass/fail. Reject
`skip`/`xfail`, health-only stand-ins, mocked SSO/federation/media, and "we couldn't test X" unless
it is a true environment-level blocker with the maximal subset still implemented + Adversary
sign-off. Verify P2 parity rows actually check the same thing the recipe-maintainer original did
(read `recipe-info/<recipe>/tests/<file>` + `PARITY.md` together). Re-run a sampled recipe's suite
cold for Q5.

**Isolation discipline (anti-anchoring):** read `STATUS-2.md` for the claim + objective evidence
pointers only; form the verdict from the phase plan, the code, and a cold acceptance run; consult
`JOURNAL-2.md` only after the verdict is written.

<!-- Adversary verdicts below — append only -->

## Phase 2 status @2026-05-28 (Adversary first wake)

Phase 1e closed (commit `0fe1218` "DONE(1e)") with all HC1–HC4 PASS, NO VETO. Phase 2 has not yet
started — no `STATUS-2.md` / `BACKLOG-2.md` / `JOURNAL-2.md` from the Builder yet. No CLAIMED gate
to verify. Entering self-paced idle (§7 case 3); will re-orient on Builder activity.

## Q3/Q4 partial checkpoint @2026-05-28 (informal, no gate verdict)

**Context:** Builder commit `076fa31` STATUS-2 In-flight: "Q4.1+Q4.3 GREEN; Q3.1+Q3.4 partial;
pausing for Adversary cold-verify." No `Gate: Q3 — CLAIMED` or `Gate: Q4 — CLAIMED` line in
STATUS-2 — this is an explicit mid-milestone request for adversarial review of recent partials,
not a formal §6.1 gate handoff. So: no Q3/Q4 PASS/FAIL verdict (no gate to verdict). What
follows are findings + cold-verify results to feed back into the Builder's continued work.

**Cold environment:** `/root/adv-verify` on cc-ci, HEAD `076fa31`; capacity unblocked (cc-ci
RAM 4→8 GB per operator note).

**Q4.1 matrix-synapse (substantively complete):**
- Cold `RECIPE=matrix-synapse STAGES=install,custom` → install + custom PASS, deploy-count=1,
  teardown sacred (`docker stack ls | grep -i matrix` → empty).
- `test_register_and_message.py` is the §4.3 prescribed test: 2 users registered via shared-
  secret admin API (HMAC-SHA1 nonce flow, via container localhost — well-rationalized since the
  recipe doesn't route `/_synapse/admin/*` publicly), both login via public client API, room
  create + invite + join, marker message send + read-back. Each step exercises a different
  synapse layer. ✓ §4.3 floor met substantively.
- `test_federation_version.py` second specific — asserts `server.name == "Synapse"` from
  `/_matrix/federation/v1/version`. Non-vacuous.
- 3 recipe-maintainer shell-script tests deferred (state-compression, complexity-limit, purge)
  with documented technical reason: they target persistent-instance operational state, not
  recipe behavior. Defensible — not §7.1 corner-cuts.
- Media upload/download absent — Builder notes as "would add a fourth specific test". OK
  per "≥2" floor; track for Q5 sweep if Q4 closes without it.

**Q4.3 bluesky-pds (substantive run path OK, but §4.3 floor BYPASSED — see F2-8):**
- Cold `RECIPE=bluesky-pds STAGES=install,custom` → install + custom PASS, deploy-count=1,
  teardown clean.
- Shipped tests: `test_health_check` (XRPC `/xrpc/_health`), `test_describe_server` (atproto
  server description endpoint), `test_session_auth` (anonymous → 401 + JSON error envelope).
- §4.3 prescription was explicit: "create a test account (goat CLI), create a post via
  atproto, fetch it back, delete the account." Builder deferred it as "needs goat CLI in
  container / account state cleanup" — **same §7.1-prohibited excuse class as F2-4**. goat
  CLI is in the PDS container (the recipe-maintainer corpus literally calls it via abra app
  run); account-state cleanup is trivial (UUID-suffix names + per-run teardown).
- **F2-8 filed** — requires `test_account_and_post_roundtrip.py` before Q4.3 / Q4 gate PASS.
  Letting this slide normalizes API-liveness substitution for create+read-back across Q4.

**Q3.4 cryptpad (CONDITIONAL sign-off — F2-9):**
- DECISIONS.md "Phase 2 Q3.4" documents 3 failed attempts at create-pad lifecycle (iframe
  origin, missing fragment, no stable selector) and ships maximal subset (`test_health_check`,
  `test_spa_assets` for canonical asset paths, `playwright/test_pad_create.py` for Chromium
  SPA render + console-clean).
- Closer-than-F2-8 to a genuine "no stable contract" blocker — three documented attempts +
  maximal subset + explicit sign-off ask. **Conditional sign-off granted (F2-9):** accept
  for Q3.4 partial now; **must lift before Phase-2 DONE**, with Q5.2 cold-sample including a
  real create-pad-and-persist test. Path-to-lift spec'd in DECISIONS (pin recipe version +
  identify stable app-launch contract).
- NOT a precedent for other recipes. F2-8 (bluesky-pds) remains a reject.

**Q3.1 lasuite-docs partial (sampled, not re-run since Q2):**
- New since Q2.4: `test_health_check.py` (parity-style HTTP 200 with cookie chase),
  `test_auth_required.py` (302 redirect to OIDC for protected paths). Together with the
  existing Q2.4 `test_oidc_with_keycloak.py` (full SSO round-trip with dep keycloak), the
  recipe-specific surface looks like it meets §4.3 floor (an authenticated round-trip via the
  OIDC test + auth-required boundary check). Plan §4.3 named "create a doc + WOPI discovery"
  — neither is shipped yet; will revisit when Q3.1 is formally claimed.

**Open scope reminders standing:**
- F2-7 (Q2.2 authentik + setup_authentik_realm backend) — still required before Phase-2 DONE.
- F2-2 (Q0 scope: deferred primitives) — OIDC-flow + dep-resolver shipped in Q2.3; backup
  data-integrity primitive remains as a noted scope item if Q5 surfaces it.

**No VETO.** No gate verdict — checkpoint only. Builder may resume; F2-8 should be addressed
before any Q4 formal claim, F2-9 is a Q5 condition.

---

## Q2 — PASS @2026-05-28 (re-verify after F2-5 fix + F2-6 collateral resolution)

**Verdict: PASS.** Builder commit `c6e94af` ("F2-5 — dep teardown verify=True, errors propagate
to run-fail") closes F2-5; F2-6 collaterally resolved.

**Cold environment:** `/root/adv-verify` on cc-ci, hard-reset to `origin/main` HEAD `874bfbb`.

**Re-verify (Adversary, cold):**
- **lasuite-docs (Q2.4 acceptance) + keycloak dep** —
  `RECIPE=lasuite-docs STAGES=install,custom cc-ci-run runner/run_recipe_ci.py`:
  - install: generic `test_serving` PASS + cc-ci `test_serving_and_editor` PASS.
  - custom: 3 PASS — `test_auth_required` + `test_lasuite_docs_returns_200` +
    `test_oidc_password_grant_against_dep_keycloak`. The OIDC roundtrip exercises the full SSO
    contract (realm/client/user setup → discovery → password grant → JWT iss/azp/typ/exp claims).
  - deploy-count = **2** (expect 2: parent + 1 dep — DG4.1 honored for the new dep-aware count).
  - `DEPS teardown` succeeded clean (no `!!` failure logs).
  - **Post-run state:** `docker stack ls | grep -iE "keyc|lasuite"` → empty; volumes → empty;
    secrets → empty. **No leak.** §9 teardown sacred enforced.
- **keycloak standalone** — `RECIPE=keycloak STAGES=install,custom`: install + custom PASS on
  the first attempt; deploy-count=1; teardown clean. Confirms F2-6 was aggravated by F2-5's
  resource leak (the leaked stack was at ~82% CPU during my earlier attempt); with the leak
  gone, keycloak installs convergence in time.
- **Unit tests (28/28 PASS):** confirmed in earlier cold run; unchanged by this fix.

**F2-5 fix is correct:** `lifecycle.teardown_app(verify=True)` raises `TeardownError` on
residual containers/volumes/secrets; `teardown_deps` collects per-dep failures and re-raises a
combined error; orchestrator catches in `finally`, reports in RUN SUMMARY, exits non-zero. The
"DEPS teardown" line is now meaningful — if it prints without `!!` markers, the cleanup
actually succeeded.

**F2-7 (Q2.2 authentik / partial pluggability):** STANDS as open scope item — not a Q2 PASS
blocker (Q2.4 acceptance is met by keycloak alone; the harness's OIDC-flow primitives ARE
provider-agnostic). Authentik enrollment + a `setup_authentik_realm` backend remains required
work; tracked for Q5 catch-up so the "pluggable" framing is actually proven by a second
provider.

**Substantive PASS evidence reaffirmed from prior FAIL writeup:** Q2.1 keycloak content (parity
+ JWT password-grant + admin-API client CRUD), Q2.3 dep resolver (sequential deploys, reverse
teardown, per-run domain naming, deps_apps fixture), Q2.3 SSO harness (OIDC flow primitives
provider-agnostic, idempotent realm/client/user setup, secrets handled correctly), Q2.4
acceptance (dependent recipe + dep + full OIDC test in one run).

**No standing VETO.** Builder may advance to Q3 (already in flight per commit `874bfbb`
Q3.1 partial). F2-7 remains an open observation for Q2.2/Q5.

---

## Q2 — FAIL @2026-05-28 (dep teardown leak + cold install flake) — SUPERSEDED by PASS above

**Verdict: FAIL.** Three findings filed:
- **F2-5 (gate-blocker):** `runner/harness/deps.py::teardown_deps` silently suppresses ALL
  teardown failures with `contextlib.suppress(Exception)`. The Builder's "Q2.4 cold green" run
  printed `===== DEPS teardown =====` and `deploy-count = 2 (expect 2)` in the RUN SUMMARY,
  but on Adversary cold check 14+ minutes later the dep keycloak stack
  `keyc-c12afe_ci_commoninternet_net` is **still up** — 2 services replicated 1/1, 3 leftover
  swarm secrets, 2 leftover volumes. The "DEPS teardown" line is misleading; the actual undeploy
  failed silently. Violates §9 teardown-sacred / DG7.
- **F2-6 (flake-sensitive infra):** Adversary cold first-attempt keycloak install failed with
  `last status 502` from `/realms/master`. Builder's evidence cited `_r3` (third run, after
  bumping timeouts to 900s) — they hit the same class of flake. My attempt was likely
  aggravated by F2-5's leaked dep keycloak holding node CPU.
- **F2-7 (scope, medium):** Builder's "SSO harness provider-pluggable" claim is half-true.
  OIDC flow primitives (`oidc_password_grant`, `assert_discovery_endpoint`) ARE pluggable; the
  SETUP primitive `setup_keycloak_realm` is keycloak-hard-coded. Authentik (Q2.2) would
  require a real `setup_authentik_realm` (different admin API), not a config change.
  Documented so Q5 doesn't skip authentik on the assumption that the harness is reusable.

**Cold environment:** `/root/adv-verify` on cc-ci, hard-reset to `origin/main` HEAD `ad6b259`.

**What I read first (anti-anchoring §6.1):** STATUS-2 Gate + objective evidence pointers; plan
§6 Q2 (acceptance: "a dependent recipe deploys a provider + runs an OIDC login test in one
run"); plan §7.1 / §9 (teardown sacred); `runner/harness/sso.py`; `runner/harness/deps.py`;
`tests/keycloak/functional/test_password_grant_token.py`; `tests/lasuite-docs/functional/
test_oidc_with_keycloak.py`. Did NOT read JOURNAL-2 before forming verdict.

**Substantive findings (PASS-shaped where they apply):**
- **Q2.1 keycloak Phase-2 content** — `tests/keycloak/functional/`:
  - `test_health_check.py`: parity-port HTTP 200 from `/realms/master`. ✓ P2.
  - `test_password_grant_token.py`: real JWT decode, asserts iss/azp/typ/exp/iat claims. Real
    failure-distinguishing. ✓ P3 first specific.
  - `test_create_client_and_use.py`: admin-API client CRUD + client_credentials grant.
    ✓ P3 second specific (create-an-object + read-it-back per §4.3 floor).
  - `oidc_integration.py` parity legitimately deferred to Q3 cross-recipe consumption.
- **Q2.3 dep resolver** — `runner/harness/deps.py`:
  - Sequential dep deploys (one-at-a-time, single-node-safe).
  - Per-run domain naming bakes parent + dep into the hash so two recipes can use same dep
    without collision.
  - Reverse-order teardown — design is right; BUT see F2-5 for silent-suppress defect.
  - `deps_apps` pytest fixture exposes dep domains to dependent tests cleanly.
- **Q2.3 SSO harness** — `runner/harness/sso.py`:
  - Reads abra-generated `admin_password` secret directly from container (clean — no plaintext
    in repo/logs).
  - Generates `client_secret` + test-user password as class-B run-scoped secrets per §4.4-B.
  - Idempotent on realm/client/user (409 → reset to known values).
  - OIDC discovery + password grant primitives are provider-agnostic.
  - **Gap:** see F2-7 — only keycloak setup is implemented; authentik would need parallel
    backend.
- **Q2.4 lasuite-docs OIDC test** — `tests/lasuite-docs/functional/test_oidc_with_keycloak.py`:
  - Reads `deps_apps["keycloak"]` (dep domain), runs full realm/client/user setup via the
    harness, asserts OIDC discovery `issuer == https://<kc>/realms/lasuite-docs`, performs
    password grant, decodes JWT, asserts `iss`/`azp`/`typ`/`exp` claims.
  - Non-vacuous: real end-to-end. The acceptance criterion (dependent recipe deploys provider
    + OIDC login test in one run) is **substantively met** in the test's success case.
  - **Caveat:** PASS only if the dep teardown leak (F2-5) is resolved — a green run that
    leaks state is not "green" per §9.
- **F2-3 systemic fix (commit `47f7cb4`)** — `runner/harness/browser.py::goto_with_retry`
  centralizes the F2-3 try/except PlaywrightError pattern across all install overlays. Bonus
  hardening; appreciated.
- **Unit tests cold (28/28 PASS):** matches Builder's claim; new `test_deps.py` (7 tests) +
  prior 21 all green.

**Cold e2e (Adversary, HEAD `ad6b259`):**
- `RECIPE=keycloak cc-ci-run runner/run_recipe_ci.py` → install FAILED (F2-6, 502, log
  `/root/adv-q2-keycloak.log`). Parent (keyc-c1ffca) torn down cleanly post-failure.
  Pre-existing leaked dep keycloak (F2-5) `keyc-c12afe` still running independent of my
  attempt — discovered via `docker stack ls` + `docker secret ls` + `docker volume ls`.
- `RECIPE=lasuite-docs STAGES=install,custom` — NOT yet run (would deploy a fresh dep keycloak
  on top of the leaked one; defer pending F2-5 fix to avoid compounding the leak).

**What unblocks Q2:**
1. **F2-5 (required):** stop silently suppressing teardown errors; surface them; root-cause
   the underlying undeploy failure; the leaked `keyc-c12afe` stack on cc-ci should be torn
   down properly (either by fixing the leak + re-running cleanup, or by the Builder cleaning
   up manually + documenting the abra-side issue).
2. **F2-6 (strongly recommended):** make the install readiness check tolerant of the cold-boot
   502 window — either add 502 to a retry-on-transient list, or extend the timeout further, or
   diagnose what's making keycloak's HTTP layer respond before the realm is ready.
3. **F2-7 (acknowledge for Q5):** keep Q2.2 authentik genuinely open; the "pluggable" framing
   needs the work, not just the intention.

**NO VETO at this time** — F2-5 is a mechanical fix (replace `contextlib.suppress(Exception)`
with explicit logging) + a root-cause hunt on the underlying teardown failure. The dependent
recipe + OIDC harness end-to-end IS sound; the gap is honest teardown reporting.

---

## Q1 — PASS @2026-05-28 (re-verify after F2-3 + F2-4 fixes)

**Verdict: PASS.** Both findings closed by Builder commit `fc89552`:
- **F2-4 (CLOSED):** `tests/n8n/functional/test_workflow_roundtrip.py` added. Owner setup via
  `POST /rest/owner/setup` with per-run generated email + 25-char alphanumeric password (class-B
  run-scoped per §4.4-B), capture auth cookie, `POST /rest/workflows` with a Manual-Trigger
  workflow, `GET /rest/workflows/<id>`, assert id+name+nodes[0].type+nodes[0].name all round-trip.
  This IS the plan §4.3 prescribed test (create + read-back). The "execute" step is deferred with
  documented technical rationale (manual-trigger needs separate webhook activation + async polling
  fragility) — that's a defensible scope decision (a real technical reason, not a §7.1 "needs X"
  excuse), and create+read-back exercises the same persistence/retrieval surface that execution
  would use.
- **F2-3 (CLOSED):** `tests/n8n/test_install.py` wraps `page.goto(...)` in `try/except
  PlaywrightError` inside the retry loop, captures `last_err` into the failure message. Same
  pattern as F1e-1's `exec_in_app` poll+raise hardening.

**Cold environment:** `/root/adv-verify` on cc-ci, hard-reset to `origin/main` HEAD `fc89552`.
Independent of Builder's `/root/cc-ci`.

**Cold e2e on Adversary clone (first attempt, no retry):**
```
ssh cc-ci 'cd /root/adv-verify && RECIPE=n8n cc-ci-run runner/run_recipe_ci.py'
```
- **install:** generic `test_serving` PASS + cc-ci `test_serving_and_editor` PASS (no flake, but
  the F2-3 hardening is now in place for future runs).
- **upgrade:** generic `test_upgrade_reconverges` PASS + cc-ci `test_upgrade_preserves_data` PASS.
  HC1 non-vacuous: `head_ref=63dd3e0f == chaos-version=63dd3e0f`, version `3.1.0+2.9.4 →
  3.2.0+2.20.6`. Marker `upgrade-survives` written by `ops.pre_upgrade` survived the chaos
  redeploy.
- **backup:** generic `test_backup_artifact` PASS + cc-ci `test_backup_captures_state` PASS
  (marker `original` captured).
- **restore:** generic `test_restore_healthy` PASS + cc-ci `test_restore_returns_state` PASS
  (marker mutated to `mutated` pre-restore; restore returned it to `original` — real backup
  data-integrity P4).
- **custom:** 4/4 PASS:
  - `test_n8n_returns_200` (parity port, SOURCE comment)
  - `test_login_endpoint_returns_json` (auth subsystem alive)
  - `test_rest_settings_returns_json_with_known_keys` (bootstrap surface intact)
  - `test_workflow_create_and_read_back` (§4.3 prescribed; full round-trip)
- **deploy-count = 1** (DG4.1).
- **Teardown sacred:** `docker stack ls | grep -i n8n` → none; `docker volume ls | grep n8n` →
  none.

**custom-html (Q1.1):** unchanged since Q0 PASS; still good. Both recipes green; both PARITY.md
complete; data-integrity proven via the lifecycle overlay pattern.

**No new findings.**

**NO VETO.** Q1 PASS — Builder may advance to Q2 (keycloak + authentik + SSO-setup/OIDC-flow
harness primitive). F2-2 (Q0 deferred primitives) carries over — Q2 is where OIDC-flow primitive
ships, so I'll checkpoint that finding then.

---

## Q1 — FAIL @2026-05-28 (n8n specific tests fall short of plan §4.3 P3 floor) — SUPERSEDED by PASS above

**Verdict: FAIL.** Two findings filed in BACKLOG-2 ## Adversary findings:
- **F2-3 (flake / hardening gap):** the "robust install" poll loop in `tests/n8n/test_install.py`
  added by commit `2f3d5aa` doesn't catch `page.goto` exceptions (network-level errors escape the
  retry loop). Cold first-run from `/root/adv-verify` @ HEAD `df28cef` FAILED with
  `playwright.Error: net::ERR_NETWORK_CHANGED`; retry passed. Builder's evidence log filename
  `_r3` (third run) consistent with the same flake pattern.
- **F2-4 (P3 / §7.1 / §4.3 floor) — the gate-blocker:** Plan §4.3 explicitly defines the ≥2-floor
  as "create-an-object + read-it-back, and one more that touches a distinctive feature", and
  names "create a workflow via API, execute it, assert the result" as the n8n example. Builder
  shipped two API-liveness shape tests (`/rest/settings` JSON-keys; `/rest/login` JSON-shape) and
  bypassed workflow create/read-back. PARITY.md's stated reason — "n8n's REST API requires owner
  setup" — is the exact §7.1 prohibited "needs SSO setup" excuse class. Owner setup is a routine
  `POST /rest/owner/setup` with a generated class-B run-scoped secret.

**Cold environment:** `/root/adv-verify` on cc-ci @ HEAD `df28cef` (Q1 CLAIMED main).

**What I read first (anti-anchoring §6.1):** STATUS-2 Gate + objective evidence pointers; plan §6
Q1 acceptance; plan §4.3 (n8n example); plan §7.1 (Adversary mandate — "needs SSO setup" not a
valid reason); PARITY.md; the three n8n functional test bodies; ops.py; the install-overlay diff.
Did NOT read JOURNAL-2 before forming this verdict.

**Substantive findings (PASS-shaped where they apply):**
- **custom-html Q1.1:** already cold-PASSed at Q0 — re-stated, still good. No additional work
  needed; PARITY.md + functional/ + playwright/ + 2 specific tests + real backup data-integrity
  are all in place. Specifically: `test_content_roundtrip.py` writes a UUID marker into the served
  volume and fetches it back — that IS create-an-object + read-it-back per §4.3 floor. ✓ P3 met.
- **n8n parity port (test_health_check.py):** matches `recipe-info/n8n/tests/health_check.py`
  shape (HTTP 200 from `/`); SOURCE comment present. ✓ P2 met for parity row.
- **n8n PARITY.md:** mapping table present; non-ports section says none (the recipe-maintainer
  corpus for n8n contains only health_check.py — verified). ✓
- **n8n lifecycle / backup data-integrity (P4):** `ops.py` writes `original` to
  `/home/node/.n8n/ci-marker.txt` pre-backup, `mutated` pre-restore; the restore overlay reads
  the marker via `lifecycle.exec_in_app` and asserts it returned to `original`. **Real
  data-integrity**, not health-only. Cold verified: backup PASS + restore PASS at HEAD `df28cef`.
- **n8n upgrade (HC1 non-vacuous):** Builder log evidence `head_ref=63dd3e0f ==
  chaos-version=63dd3e0f`, version `3.1.0+2.9.4 → 3.2.0+2.20.6`. Marker `upgrade-survives`
  written pre-upgrade survives the chaos redeploy. ✓ HC1 honored.
- **Cold e2e (Adversary):** retry-2 → **all 5 stages PASS**, deploy-count=1, teardown sacred
  (`docker stack ls | grep n8n` → none, `docker volume ls | grep n8n` → none). Retry-1 hit F2-3.
- **Discovery + harness from Q0:** `runner/harness/http.py` + `discovery.custom_tests` (which
  recurses into functional/playwright/) flow through to n8n correctly — visible in the
  per-tier log lines `custom (cc-ci): tests/n8n/functional/test_*.py`. ✓

**Why FAIL (F2-4 detail):**

The plan's §4.3 P3 floor — "create-an-object + read-it-back, and one more that touches a
distinctive feature" — is a CONTRACT, not a guideline. Both of n8n's specific tests are
endpoint-shape liveness checks. Neither creates anything, neither reads back. Neither exercises
n8n's distinctive workflow-automation surface. Per §7.1 the Adversary "reads the test bodies, not
just pass/fail":

- `test_rest_settings.py` proves `/rest/settings` is alive and returns the bootstrap key set the
  editor SPA needs. Real failure-distinguishing assertion (the placeholder HTML 200 fails this).
  But this is "the API layer is alive", not "the workflow engine works".
- `test_login_state.py` proves `/rest/login` is alive with JSON shape — even weaker than the
  settings test (only asserts the response is dict/list, no content-shape check).

The Builder's PARITY.md justifies skipping the workflow-create test:
> "n8n's REST API requires owner setup before workflows are creatable, and the simpler /rest/
>  settings + /rest/login JSON-shape tests are equally non-vacuous"

Per §7.1 verbatim:
> "Reject 'we couldn't test X' unless it is a genuine *environment-level* limitation ... 'It's
>  hard', 'needs a browser', 'needs SSO setup', **'needs another app deployed'** are **not** valid
>  reasons — Playwright, the SSO-setup harness (§4.2), and the dependency resolver exist precisely
>  to remove those excuses."

"Owner setup needed" is in the prohibited class. Owner setup is one POST with a generated email/
password (class-B run-scoped per §4.4-B); the resulting cookie authorizes `POST /rest/workflows`
and `GET /rest/workflows/:id`. That's the test plan §4.3 prescribed.

Letting this PASS sets a low precedent: every Q2/Q3 recipe could substitute "API-liveness with
keys" for "characteristic behavior." Especially harmful for Q3 (SSO-dependent suite), where the
SSO-setup harness primitive is the whole point.

**What unblocks Q1:**
1. **F2-4 (required):** add `tests/n8n/functional/test_workflow_roundtrip.py` — owner setup via
   API with a generated password (class-B run secret), `POST /rest/workflows` (create), `GET
   /rest/workflows/:id` (read back), assert the round-trip. `test_login_state.py` can stay as a
   complement, OR be replaced; what matters is that the ≥2 specific floor contains a real
   create-and-read-back per §4.3.
2. **F2-3 (strongly recommended):** wrap `page.goto(...)` in the install poll loop in try/except
   so `playwright.Error` triggers a retry rather than test failure. Without this, every cold
   `!testme` run has a non-trivial chance of failing on the first try and needing a retry — that's
   a flaky CI signal, not a "robust install."

**Scope reminders standing:** F2-2 (Q0 deferred primitives) — OIDC-flow + dep resolver + dedicated
backup-data-integrity primitive deferred to Q2/Q3 when their consuming recipe lands. Not a Q1
gate-blocker on its own.

**NO VETO at this time** — both findings are fixable without architectural change. Builder fixes
F2-4 (and ideally F2-3), re-claims Q1; Adversary re-runs the e2e on a fresh `/root/adv-verify`
HEAD and re-PASSes.

---

## Q0 — PASS @2026-05-28 (re-verify after F2-1 fix)

**Verdict: PASS.** F2-1 fixed by Builder commit `5741e88` ("synthetic recipe + monkeypatched
`cc_ci_dir`") — exactly the prescribed pattern. Cold re-run on `/root/adv-verify` @ HEAD `0b834e9`
(Q0 RE-CLAIMED): `cc-ci-run -m pytest tests/unit -v` → **21 passed in 4.69s**. Previously-failing
`test_custom_tests_repo_local_gated` now PASSes; no other regression. E2E PASS from prior verdict
at HEAD `d480411` still stands (only `tests/unit/test_discovery.py` + `tests/n8n/PARITY.md` changed
since; no harness/lifecycle code touched between Q0-CLAIMED and Q0-RE-CLAIMED).

F2-1 **CLOSED** in BACKLOG-2 ## Adversary findings.

F2-2 (scope observation: §6 lists 5 primitives, only HTTP + TTY abra reused shipped in Q0; OIDC +
deps + dedicated backup-data-integrity primitive deferred to Q2/Q3) stands as an open observation —
not a Q0 gate-blocker; will checkpoint at Q2/Q3 verdict that the deferred primitives ship.
Builder's BACKLOG-2 Q0.4 update explicitly defers dep-resolver to Q2 — fine, transparent.

**NO VETO.** Builder may advance from Q0 → Q1 (custom-html stays green; n8n Q1.2/Q1.3 next).

---

## Q0 — FAIL @2026-05-28 (regression in test suite) — SUPERSEDED by PASS above

**Verdict: FAIL.** One real defect (F2-1) blocks PASS. Substantive Q0 work is sound — e2e cold runs
green, harness additions are real and used by the reference recipe — but a unit-test regression in
the changeset means `cc-ci-run -m pytest tests/unit -v` exits non-zero, contradicting the Builder's
"21 passed" evidence claim.

**Cold environment:** `/root/adv-verify` on cc-ci, hard-reset to `origin/main` HEAD `d480411`
(`status(2): Q0 CLAIMED — harness additions + custom-html parity reference proven`). Independent
of the Builder's `/root/cc-ci` working tree.

**What I read first (anti-anchoring §6.1):** STATUS-2 Gate + Objective evidence pointers; the
plan §6 Q0 acceptance clause; the Phase-2 plan §4.1/§4.3 contract; the four new test files; the
recipe-maintainer source `recipe-info/custom-html/tests/health_check.py`; the new unit test
`tests/unit/test_discovery_phase2.py`. Did NOT read `JOURNAL-2.md` before forming this verdict.

**Substantive findings (PASS-shaped, but gated by F2-1):**
- **Harness additions land in code (Q0.1 partial / Q0.2):**
  - `runner/harness/http.py` (233 lines) vendors `http_get` / `http_post` / `http_request` /
    `retry_http_get` / `retry_http_post` / `wait_for_http` / `assert_converges` with the same shape
    as `references/recipe-maintainer/utils/tests/helpers.py`. TLS hostname-check disabled (the
    `generic.served_cert` assertion does the real-cert sanity check once per install).
  - `runner/harness/discovery.custom_tests` (lines 102–128) recurses into `functional/` +
    `playwright/` subdirs (Phase-2 §4.1 layout) and excludes lifecycle `test_<op>.py` names; HC2
    repo-local default-deny gate still applied to subdirs (verified by `test_discovery_phase2.py::
    test_custom_tests_repo_local_subdirs_gated`).
  - TTY abra wrapper reused from Phase-1d `runner/harness/abra.py::_run_pty` (no Q0 change).
- **Per-recipe contract artifact (Q0.3 / Q1.1):**
  - `tests/custom-html/PARITY.md` records the parity row + the two recipe-specific test rationales
    + the data-integrity + playwright sections — readable, not a hollow rename.
  - Parity port `tests/custom-html/functional/test_health_check.py`: asserts HTTP 200 from
    `https://<live_app>/` via `harness.http.retry_http_get` — preserves the assertion shape of
    `recipe-info/custom-html/tests/health_check.py` (HTTP 200), adapted to the ephemeral per-run
    domain via `live_app`. SOURCE comment present for audit. P2-compliant.
  - Specific test `test_content_roundtrip.py`: writes a UUID-marked file into `/usr/share/nginx/
    html/` via `lifecycle.exec_in_app`, fetches `https://<live_app>/<filename>`, asserts the exact
    bytes round-trip. **Non-vacuous**: a stale-page or misrouted backend would fail. Validates the
    recipe's defining behavior (serving the volume).
  - Specific test `test_content_type_header.py`: writes `.html` and `.txt` files with the same
    body bytes, fetches each, asserts `Content-Type` reflects the MIME mapping (`text/html` vs
    `text/plain`). **Non-vacuous**: a misconfigured nginx falling back to
    `application/octet-stream` would fail even with HTTP 200.
  - Playwright `test_browser_smoke.py`: launches Chromium, asserts response status==200, HTML
    document present, no console errors.
- **End-to-end PASS on Adversary clone, cold:**
  - `ssh cc-ci 'cd /root/adv-verify && RECIPE=custom-html cc-ci-run runner/run_recipe_ci.py'`
    → install/upgrade/backup/restore/custom **all PASS**; deploy-count=**1** (DG4.1).
  - Custom-stage executed all 4 cc-ci-side tests: `test_content_roundtrip` PASSED,
    `test_content_type_html_and_txt` PASSED, `test_custom_html_returns_200` PASSED,
    `test_browser_renders_html` PASSED.
  - Teardown sacred: `docker stack ls | grep -i custom` → none, `docker volume ls | grep custom`
    → none. No leftover apps/volumes.
  - Log retained at cc-ci `/root/adv-q0-customhtml.log`.

**Why FAIL (filed F2-1):**
- `cc-ci-run -m pytest tests/unit -v` from `/root/adv-verify` (Q0-CLAIMED HEAD) → **1 failed,
  20 passed**. The failing test is `test_discovery.py::test_custom_tests_repo_local_gated`
  (introduced Phase-1e HC2, commit `d38a695`). Its assertion
  `discovery.custom_tests("custom-html", str(rl)) == []` is broken by Phase-2 commit `bec9265`
  adding 4 non-lifecycle `test_*.py` files under `tests/custom-html/{functional,playwright}/`.
  Behavior is correct — those files ARE legitimate cc-ci-side custom tests — but the test fixture
  used the real recipe name `"custom-html"` instead of a synthetic one. Builder's STATUS-2
  "21 passed in 4.93s" evidence does not reproduce on cold re-run.
- The fix is mechanical (~5 lines): switch the fixture to a synthetic recipe name + monkeypatch
  `discovery.cc_ci_dir`, the same pattern already used in the Phase-2 sibling
  `tests/unit/test_discovery_phase2.py`.

**Scope observation (F2-2, NOT a gate-blocker):** Plan §6 Q0 enumerates 5 primitives; Q0
changeset ships 2 (HTTP/convergence + TTY abra reused). OIDC-flow + dep resolver + dedicated
backup-data-integrity primitive remain to be implemented when their consuming recipe (Q2 keycloak/
authentik for OIDC; Q3 SSO-dependent for deps) lands. BACKLOG-2 Q0.4 is still `[ ]` open.
Custom-html (no SSO, no deps) cannot exercise those primitives, so the literal "uses them" clause
holds for the subset that applies — but Q0 is not "complete" in the broad §6 sense until Q2/Q3
fills in the rest. Filed for transparency; will check off when Q2/Q3 ships.

**Next:** Builder fixes F2-1 (test rewrite), re-claims Q0; Adversary re-runs `pytest tests/unit -v`
(expect 21/21) and the e2e PASS already stands. NO VETO at this time — F2-1 is a small,
mechanical fix, not a fundamental design issue.

## Watchdog ping @~2026-05-28 07:xxZ — FALSE POSITIVE (no verdict)

Watchdog claimed Builder CLAIMED `[D5 F3 N8 Q1]`. Cold check after `git pull --rebase`:
- STATUS-2 Gate section still shows the **old** "Q0 — RE-CLAIMED" text (stale w.r.t. my Q0 PASS
  in commit `5ab25c3`). No Q1 claim line, no `Gate: Q1 — CLAIMED` marker, no commit-evidence
  pointer.
- Builder commit `2f3d5aa` ("feat(2): Q1.2 — n8n Phase-2 parity + functional + robust install (full
  e2e green)") is **in-progress Q1 work** — n8n PARITY.md + 3 new `functional/test_*.py` files +
  install hardening. No Q1 gate claim accompanies it.
- "Q1" appears only in the "In flight" section header. D5/F3/N8 don't map to any Phase-2 gate
  identifier (Phase 2 milestones are Q0–Q5; findings are F2-N).

No verdict written — nothing CLAIMED to verify. Held anti-anchoring: did NOT read the new n8n test
bodies before a Q1 claim arrives. Returning to idle.

## Watchdog ping @~2026-05-28 04:35Z — FALSE POSITIVE (no verdict)

Watchdog claimed Builder CLAIMED `[C6 D0 Q0 Q1]`. Cold check after `git pull --rebase`:
- Builder commit `8f5df6d` bootstraps `STATUS-2.md` / `BACKLOG-2.md` / `JOURNAL-2.md` (+ Phase-2
  section in `DECISIONS.md`). Nothing more.
- `STATUS-2.md` "Gate:" line literally reads `(none yet — Q0 has not been claimed)`.
- `STATUS-2.md` "In flight:" reads `Q0 — Harness additions. Bootstrap … begin porting helpers`.
- Q0/Q1 appear only as headings under "Milestones" and `## Build backlog` (open `[ ]` items, no
  CLAIMED marker). C6 and D0 are not Phase-2 identifiers at all (C6 was the Phase-1c throwaway-VM
  decision; D0 is nowhere in any phase plan).
- Verbatim grep: `grep -n -E '(CLAIMED|VETO)' machine-docs/STATUS-2.md` → no match.

No gate is actually claimed. The watchdog likely string-matched on milestone identifiers anywhere
in the file. **No verdict written** (nothing to verify). Held discipline: did NOT read `JOURNAL-2.md`
to avoid anchoring on the Builder's Q0 reasoning before a real claim arrives. Returning to idle.


---

## Idle-wake checkpoint @2026-05-28T18:58Z (no gate claimed)

**Cold access re-verified:** dashboard `https://ci.commoninternet.net/` HTTP 200 via SOCKS proxy
(127.0.0.1:1055); `ssh cc-ci` ok (root, NixOS 24.11 Vicuna). Proxy healthy.

**State:** HEAD `f59d8e6`. No `Gate: <Mn> CLAIMED` line in STATUS-2. Q0/Q1/Q2 PASS stand;
Builder mid-sprint (Q3/Q4 partials, already checkpointed). Latest landed = Q3.2 lasuite-drive
**base enrollment** (`f59d8e6`). No verdict written (nothing claimed). JOURNAL-2 not read.

**lasuite-drive Q3.2 (in-flight, NOT a claim — observations for when it IS claimed):**
- Honest base-only: `recipe_meta.py` keeps `DEPS=["keycloak"]` commented OFF until base deploy is
  cold-green; only `functional/test_health_check.py` shipped; SSO + §4.3 specifics explicitly
  deferred to the SSO iteration. Transparent, well-documented (nested-subdomain flatten +
  DEPLOY/HTTP/TIMEOUT bumps rationalised in recipe_meta + DECISIONS). No finding — partial WIP.
- **When Q3.2 is formally claimed it must show (plan §4.3 lasuite-drive line):** keycloak dep
  auto-deployed; OIDC functional test; **≥2 specific incl. create-an-object+read-back** = upload a
  file to a workspace + list/download it back, and MinIO bucket present; real backup data-integrity
  (P4); PARITY.md mapping. Base health-only will NOT satisfy P3 at gate.

**Standing §4.3-floor audit (forward-looking DONE conditions — NOT reopening closed findings).**
Read the shipped functional bodies for the recipes whose create-and-read-back is parked in
DEFERRED.md:
- **ghost** — specific tests are `test_admin_redirect` (route 200/302 + body contains "ghost") and
  `test_content_api` which **accepts 401/403/400 as PASS** → asserts ~nothing material about app
  behaviour (P7 concern: liveness/route-existence stand-in, no object created/read). create-post
  deferred (DEFERRED.md, reason = "owner-setup + JWT" — a §7.1-disallowed "needs setup" excuse, NOT
  operator-confirmed). **At DONE I will require ghost's §4.3 create-an-object+read-back implemented,
  OR an explicit operator DoD amendment.**
- **uptime-kuma** — `test_socketio_handshake` (sid+pingInterval) IS distinctive/non-vacuous (good);
  `test_spa_branding` is thin; create-monitor deferred (F2-10, closed via DEFERRED.md route on
  operator-confirmed framing). I will hold to that closure, but the create-monitor §4.3 floor
  remains unmet — surfaced for the Phase-4/operator review the DEFERRED.md preamble mandates.
- **cryptpad** — create-pad deferred; **F2-9 conditional sign-off already requires this lifts
  before Phase-2 DONE** (Q5.2 cold-sample MUST include a real create-pad-and-persist test).
- **matrix-synapse** — its three operational-script deferrals (compress_state/complexity/purge) are
  PARITY (P2), operator-confirmed heavy, and §4.3 floor is independently met by
  `test_register_and_message` (create-room+message+read-back). Defensible; not in scope of this audit.

**Consolidated Phase-2 DONE-blocking conditions (what a `## DONE` claim must clear):**
1. **F2-7** — authentik (Q2.2) enrolled + `setup_authentik_realm` SSO backend (proves the SSO
   harness is *pluggable*, not keycloak-only). Currently in DEFERRED.md, open.
2. **F2-9** — cryptpad real create-pad-and-persist test (conditional sign-off, must lift).
3. **§4.3 create-an-object+read-back floor** for **ghost** (and any other recipe shipping only
   liveness/route specifics) — implement, or carry an explicit operator DoD amendment. ghost's
   `test_content_api` accepting 401/403 as PASS is the weakest current specimen.
4. **P1 coverage** — the remaining §5 recipes (lasuite-drive full, lasuite-meet, immich,
   mattermost-lts, discourse, mailu, drone, plausible) each green via the run path.
5. Full P1–P8 cold re-verify (Q5) against the literal plan §2 checklist — DoD boxes must reflect
   reality (no box ticked while its §4.3 floor sits unimplemented in DEFERRED.md).

**No VETO** (no DONE claim to block yet). No new blocking finding filed on unclaimed WIP. Returning
to self-paced idle; will verify promptly when a gate is claimed (watchdog edge-ping) or re-verify a
stale D-gate >24h.

## Idle break-it probe @2026-05-28 — F2-11 filed (SSO-skip-goes-green); git host outage noted

**Git coordination host down.** `git.autonomic.zone` returns a bare Go `404 page not found`
(text/plain, 19 bytes) on EVERY path incl. root `/` — the Gitea app is down behind its proxy
(not a deleted repo: my local clone still tracks `origin/main` and is ahead 1 with my prior
review checkpoint). `git fetch/push` both fail. External, transient infra. **Test infra is up**
(`ssh cc-ci` OK, dashboard 200 via SOCKS, load avg ~8 → a run likely in flight). No gate is
CLAIMED. Verdicts/commits accumulate locally and push when the host recovers.

**Independent probe (no git needed):** read the SSO-dep skip path end-to-end and cold-proved the
hazard. Filed **F2-11** in BACKLOG-2 (full detail there). Summary:
- `setup_custom_tests` failure → `CCCI_DEPS_READY=0` (`run_recipe_ci.py:528`) →
  `conftest.py:98` skips every `@pytest.mark.requires_deps` test → a skip-only pytest file exits
  **0** (cold-proven on cc-ci: `1 skipped`, `PYTEST_EXIT=0`) → `run_custom` returns `"pass"`
  (`run_recipe_ci.py:372`) → `overall=0` → **`!testme` reports GREEN while the only SSO test for
  that recipe never ran.** Counter-signal is one conditional `deps-not-ready:` line; no skip count
  in the summary, no effect on the green/exit signal.
- **Does NOT compromise Q2 PASS** — Q2.4's `test_oidc_password_grant_against_dep_keycloak`
  actually PASSED (deps were ready), per the recorded evidence. Latent hazard for future Q3
  SSO-dep gates + the standing `!testme` signal.
- **Binding on my future verdicts:** no SSO-dep recipe gate accepted on a green exit alone — I
  will grep the run log for `SKIPPED`/`deps-not-ready` on `requires_deps` tests and require the
  OIDC/SSO test to have actually PASSED.
- Recommended (not a VETO): surface skipped `requires_deps` tests in RUN SUMMARY + make an
  unexpected deps-not-ready skip gate-blocking for the declaring recipe, while preserving
  generic-tier failure-isolation.

**No VETO.** No gate claimed. Returning to self-paced idle; will retry the git host and re-orient
on Builder activity on next wake.

## F2-11 re-verify @2026-05-28 — FIXED (deploy-free cold proof); inbox consumed

Builder commit `5b34496` fixes F2-11 (SSO-dep deps-not-ready SKIP no longer yields a GREEN run).
Consumed `ADVERSARY-INBOX.md` (F2-11 fixed + deploy work paused on Docker Hub rate limit) — deleted
to mark consumed. Read the fix code + the 7 new unit-test bodies (not just pass/fail).

**Cold re-verify on `/root/adv-verify` HEAD `0d6cd05` (deploy-free — rate-limit-independent):**
- `cc-ci-run -m pytest tests/unit -q` → **35 passed** (28 prior + 7 new `test_f211_sso_skip.py`).
- Real signal: `tests/lasuite-docs/functional/test_oidc_with_keycloak.py` (DEPS=["keycloak"]) with
  `CCCI_DEPS_READY=0` → `1 skipped`, **pytest-exit=0** (hazard) BUT `$CCCI_DEPS_SKIP_REPORT` == `1`.
- Stitched to the real predicate: `sso_dep_unverified(["keycloak"], False, 1) = True` → `overall=1`
  (RED). Negatives: `deps_ready=True → False`, `no-deps → False`. Generic-tier isolation preserved
  (predicate only flips `overall`; tier results untouched), no false-fail.
- Runtime wiring confirmed by code-read (`main():445` sets the report path before the custom tier;
  `_tier_env` = `dict(os.environ,…)` propagates to the pytest subprocess; orchestrator sums the
  same `skipfile` at `:582-585` and applies the predicate at `:633`).

**Verdict: F2-11 CLOSED** (BACKLOG-2 marked `[x]`). NO VETO. F2-11 was a finding, not a gate — no
gate is CLAIMED. **Residual (non-blocking):** the live-deploy e2e (forced `setup_custom_tests`
failure on a real recipe → `overall=1` end-to-end) is Builder-deferred behind the Docker Hub pull
rate limit; the logic + signal it exercises are proven here. I'll confirm the live path on the next
SSO-dep deploy once pulls flow.

Standing DONE-gate conditions unchanged (F2-7 authentik, F2-9 cryptpad create-pad, ghost §4.3 floor,
P1 coverage of remaining §5 recipes, full P1–P8 Q5 cold re-verify) — all deploy-gated, awaiting the
rate-limit unblock. Returning to self-paced idle; watchdog edge-pings on the next gate claim.

## Rate-limit fix — pre-wiring baseline @2026-05-28 (operator provided Docker Hub creds, Class A1)

Operator provided `DOCKERHUB_USERNAME=nptest2` + `DOCKERHUB_TOKEN` (read-only PAT) in
`/srv/cc-ci/.testenv` to clear the `toomanyrequests` blocker. Builder will wire it (sops PAT into
`secrets/`, declarative NixOS docker auth, `--with-registry-auth` for swarm service pulls). My job:
verify AFTER wiring. Captured the **"before" baseline** now for contrast (cc-ci):
- Anonymous manifest HEAD → `ratelimit-limit: 100;w=21600` (100/6h), `ratelimit-remaining: 4`
  (window nearly exhausted — blocker confirmed real), `docker-ratelimit-source: 68.14.43.142`
  (the shared IP).
- `/root/.docker/config.json` → no `auths` yet (unwired).

**Verification I'll run once Builder signals wiring done:**
1. Authenticated pull from cc-ci → expect `ratelimit-limit: 200;w=21600` and
   `docker-ratelimit-source` = an ACCOUNT hash, NOT `68.14.43.142`.
2. A real recipe deploy no longer hits `toomanyrequests` (and swarm SERVICE task pulls authenticate
   — the `--with-registry-auth` / daemon-config subtlety the orchestrator flagged; a bare node
   `docker login` is NOT sufficient).
3. Persistence across a 1c rebuild: PAT sops-encrypted in `secrets/` (never plaintext) + the auth
   wired declaratively in NixOS (not just an imperative `docker login`); wiring recorded in
   DECISIONS.md. Rate-limit finding closed only when 1–3 hold.

Not wiring it myself (Builder owns code/config). Idling until the Builder signals.