diff --git a/machine-docs/BACKLOG-2.md b/machine-docs/BACKLOG-2.md index 53d379d..5474ebd 100644 --- a/machine-docs/BACKLOG-2.md +++ b/machine-docs/BACKLOG-2.md @@ -115,6 +115,62 @@ Phase plan: `/srv/cc-ci/cc-ci-plan/plan-phase2-recipe-tests.md` ## Adversary findings +- [ ] **F2-11 [adversary] — SSO-dep "deps-not-ready" SKIP yields a GREEN `!testme` while the + core OIDC test never ran (gate-integrity / P7, medium)** — Filed by Adversary @2026-05-28 + as an independent break-it probe during the git.autonomic.zone outage (no gate claimed). + + **The hazard chain (cold-proven, end-to-end):** + `runner/run_recipe_ci.py:516` — if the `setup_custom_tests` step raises (dep deploy / SSO + realm enrich / hook redeploy fails), it sets `deps_ready=False` and *does not abort the run* + (by design — failure-isolation). At line 528 it exports `CCCI_DEPS_READY=0`. Then + `tests/conftest.py:98-112` (`pytest_collection_modifyitems`) adds a + `pytest.mark.skip(reason="deps-not-ready: …")` to every `@pytest.mark.requires_deps` test — + which for an SSO-dependent recipe is the ONLY meaningful test (e.g. lasuite-docs + `test_oidc_with_keycloak.py`, `test_oidc_login.py`, `test_create_doc.py` are all + `requires_deps`). A pytest file whose only test is skipped exits **0**: + - Cold-proven on cc-ci @2026-05-28: a one-test file marked + `@pytest.mark.skip(reason="deps-not-ready: …")` → `1 skipped in 0.01s`, `PYTEST_EXIT=0`. + - `run_custom` (`run_recipe_ci.py:372`) returns `"pass"` whenever `rc==0`, so the custom + tier is `pass`. The RUN SUMMARY (`overall`, lines 587-603) flips to `1` only on + deploy-count mismatch, dep-teardown leak, a tier == `"fail"`, or no-tiers. A skip is none + of those → **`overall=0` → the run reports fully GREEN.** + - The only counter-signal is a single ` deps-not-ready: ` line, printed *only* + `if not deps_ready` (line 581-582), with NO skip count in the per-tier summary and no + change to the green/exit signal. + + **Why it matters (P7 / §7.1):** for any SSO-dependent recipe, a green `!testme` would then + mean "generic install/upgrade/backup passed" while the characteristic OIDC/SSO test — the + whole point of P2/P3/P6 coverage for that recipe — silently skipped. P7 forbids a skip that + lets a recipe go green. The design's failure-isolation (don't let a transient SSO outage + break the generic-tier signal) is legitimate; the defect is that the *green run signal* is + indistinguishable from "SSO verified," and nothing makes an unexpected SSO-test skip + gate-blocking or even loudly visible in the summary. + + **Did NOT compromise the existing Q2 PASS:** Q2.4 evidence (STATUS-2 + my REVIEW-2 Q2 PASS) + shows `test_oidc_password_grant_against_dep_keycloak` actually **PASSED** (`1 PASS`), not + skipped — deps_ready was true. So Q2 stands. This is a latent hazard for every *future* + SSO-dep gate (Q3 lasuite-*/immich/cryptpad-with-deps) and for the standing `!testme` signal. + + **Adversary acceptance-discipline (binding on me, effective now):** I will NOT accept any + SSO-dependent recipe's gate on a green exit alone. For Q3 and any deps-declaring recipe I + must grep the run log for `SKIPPED` / `deps-not-ready` on `requires_deps` tests and require + the OIDC/SSO test to have actually **PASSED**. A skipped core test = NOT a PASS, regardless + of `overall=0`. + + **Recommended Builder fix (not a VETO; no SSO-dep gate is claimed right now):** + 1. Surface skipped `requires_deps` tests in the RUN SUMMARY — e.g. a per-tier + `custom: pass (N skipped: deps-not-ready)` and an explicit `!! N requires_deps tests + SKIPPED — SSO unverified` warning line. + 2. Make an *unexpected* deps-not-ready skip gate-blocking: when a recipe declares `DEPS` and + `setup_custom_tests` fails, the run should not be reported as a clean PASS for that + recipe (e.g. `run_custom` could distinguish skip-only-of-required-tests from genuine + pass, or the orchestrator could set `overall=1` when `not deps_ready` and any + `requires_deps` test was thereby skipped). Failure-isolation for the *generic* tiers can + be preserved while still failing the recipe's own SSO claim. + - Repro: set `CCCI_DEPS_READY=0` (or force a `setup_custom_tests` raise) and run any + deps-declaring recipe through `runner/run_recipe_ci.py` with `STAGES=install,custom`; + observe `custom: pass` + `overall=0` while the OIDC test shows `SKIPPED`. + - [x] **F2-10 [adversary] — CLOSED @2026-05-28 via Builder route 2** (file in DEFERRED.md per the new orchestrator-confirmed convention). The uptime-kuma create-a-monitor entry is in `machine-docs/DEFERRED.md` (commit `650ab47` migrated + `44e88f3` relocated under Open diff --git a/machine-docs/BUILDER-INBOX.md b/machine-docs/BUILDER-INBOX.md index c0910a9..6131120 100644 --- a/machine-docs/BUILDER-INBOX.md +++ b/machine-docs/BUILDER-INBOX.md @@ -19,3 +19,18 @@ detail in REVIEW-2 "Idle-wake checkpoint @2026-05-28T18:58Z": reality (no box ticked while its §4.3 floor sits in DEFERRED.md). No action required now — these bite only at gate/DONE. Carry on. (Delete this file once read.) + +## @2026-05-28T19:40Z — F2-11 filed (SSO-skip-goes-green) + git host outage + +- **git.autonomic.zone is down** (bare Go `404` on every path incl. root — Gitea app down behind + its proxy). I can't push verdicts/findings until it recovers; this note + F2-11 + REVIEW-2 + entry are committed locally and will arrive together when the host is back. Test infra (cc-ci) + is unaffected. +- **F2-11 [adversary], medium, NOT a VETO** (full detail in BACKLOG-2 ## Adversary findings): + when `setup_custom_tests` fails for a DEPS-declaring recipe, `CCCI_DEPS_READY=0` makes every + `@pytest.mark.requires_deps` test SKIP, a skip-only pytest file exits 0, `run_custom` returns + `"pass"`, `overall=0` → **`!testme` goes fully GREEN while the recipe's only SSO/OIDC test never + ran.** Cold-proven on cc-ci. Does NOT affect the Q2 PASS (that OIDC test actually passed). + Suggested fix: surface skipped `requires_deps` tests in RUN SUMMARY + make an unexpected + deps-not-ready skip gate-blocking for the declaring recipe (keep generic-tier isolation). I will + not accept any SSO-dep gate on a green exit alone — the OIDC test must have PASSED, not skipped. diff --git a/machine-docs/REVIEW-2.md b/machine-docs/REVIEW-2.md index 3444bc0..19c73de 100644 --- a/machine-docs/REVIEW-2.md +++ b/machine-docs/REVIEW-2.md @@ -571,3 +571,33 @@ DEFERRED.md: **No VETO** (no DONE claim to block yet). No new blocking finding filed on unclaimed WIP. Returning to self-paced idle; will verify promptly when a gate is claimed (watchdog edge-ping) or re-verify a stale D-gate >24h. + +## Idle break-it probe @2026-05-28 — F2-11 filed (SSO-skip-goes-green); git host outage noted + +**Git coordination host down.** `git.autonomic.zone` returns a bare Go `404 page not found` +(text/plain, 19 bytes) on EVERY path incl. root `/` — the Gitea app is down behind its proxy +(not a deleted repo: my local clone still tracks `origin/main` and is ahead 1 with my prior +review checkpoint). `git fetch/push` both fail. External, transient infra. **Test infra is up** +(`ssh cc-ci` OK, dashboard 200 via SOCKS, load avg ~8 → a run likely in flight). No gate is +CLAIMED. Verdicts/commits accumulate locally and push when the host recovers. + +**Independent probe (no git needed):** read the SSO-dep skip path end-to-end and cold-proved the +hazard. Filed **F2-11** in BACKLOG-2 (full detail there). Summary: +- `setup_custom_tests` failure → `CCCI_DEPS_READY=0` (`run_recipe_ci.py:528`) → + `conftest.py:98` skips every `@pytest.mark.requires_deps` test → a skip-only pytest file exits + **0** (cold-proven on cc-ci: `1 skipped`, `PYTEST_EXIT=0`) → `run_custom` returns `"pass"` + (`run_recipe_ci.py:372`) → `overall=0` → **`!testme` reports GREEN while the only SSO test for + that recipe never ran.** Counter-signal is one conditional `deps-not-ready:` line; no skip count + in the summary, no effect on the green/exit signal. +- **Does NOT compromise Q2 PASS** — Q2.4's `test_oidc_password_grant_against_dep_keycloak` + actually PASSED (deps were ready), per the recorded evidence. Latent hazard for future Q3 + SSO-dep gates + the standing `!testme` signal. +- **Binding on my future verdicts:** no SSO-dep recipe gate accepted on a green exit alone — I + will grep the run log for `SKIPPED`/`deps-not-ready` on `requires_deps` tests and require the + OIDC/SSO test to have actually PASSED. +- Recommended (not a VETO): surface skipped `requires_deps` tests in RUN SUMMARY + make an + unexpected deps-not-ready skip gate-blocking for the declaring recipe, while preserving + generic-tier failure-isolation. + +**No VETO.** No gate claimed. Returning to self-paced idle; will retry the git host and re-orient +on Builder activity on next wake.