# cc-ci Phase 3r — `/ci-test-review` Claude skill (on-demand AI review + diagnosis)

**Status:** QUEUED — after Phase 3 (results UX), before Phase 4 (final review). A **Claude skill that
ships in the cc-ci product repo** (`.claude/skills/ci-test-review/`), built by the loops.
**Transition:** auto (in the launcher sequence). **Owner:** Builder + Adversary loops.
**This file:** `/srv/cc-ci/cc-ci-plan/plan-phase3r-ci-test-review-skill.md`
**Phase order:** … 3 → **3r** → 4.

---

## 0. Why — two distinct modes

cc-ci's **primary** use is **deterministic testing with NO AI**: `!testme` / the nightly sweep run the
harness, recipes pass/fail, done. That stays the bread-and-butter and is unaffected by this phase.

This phase adds a **separate, on-demand, AI-driven REVIEW layer** — the `/ci-test-review` skill — that
a maintainer (or a fresh Claude) invokes to: run the full suite across **all** recipes, and **for any
failure, diagnose the root cause and classify it** — does it need a **PR to the recipe** (and what
that PR is), a **PR to the CI server** (harness/test/infra), or is the **test out of date**? — or, if
everything's green and current, report **"all passed, recipes + tests up to date."** It codifies the
recipe-PR-vs-CI-PR judgment the Adversary already exercises (e.g. lasuite-drive→recipe PR, immich
pg_dump→recipe PR) into a runnable tool.

**Crisp boundary:** the test *execution* is deterministic (the existing harness); **AI is used ONLY
for diagnosis/classification/PR-proposal**, never to decide pass/fail.

## 1. Definition of Done (Adversary cold-verifies → `machine-docs/REVIEW-3r.md`)

- [ ] **R1 — Skill exists + invocable.** `.claude/skills/ci-test-review/SKILL.md` (+ helper script)
      in the cc-ci repo, invocable as `/ci-test-review`. Reuses the existing harness + recipe
      enrollment list — does NOT reinvent the runner.
- [ ] **R2 — Deterministic execution.** Runs the real harness (`run_recipe_ci.py`) across **all
      enrolled recipes**, full suite per recipe, collecting per-recipe / per-tier pass/fail/skip.
      No AI in execution. (Decide cold vs `--quick`: default **cold** for an authoritative review;
      `--quick` as a fast-mode option.)
- [ ] **R3 — "Up to date" freshness.** Tests each recipe at its **latest published version** (not a
      stale pin) and flags any recipe whose tested version lags upstream, and any test that's stale
      vs the recipe — so "all passed" means "passes against current upstream," not against an old pin.
- [ ] **R4 — All-green output.** If every recipe passes AND is current, emit a clear **"ALL PASSED —
      recipes up to date, tests up to date"** summary (with versions tested).
- [ ] **R5 — Failure diagnosis + classification.** For each failure, AI determines root cause and
      classifies it as exactly one of:
      - **RECIPE bug** → needs a **recipe PR**; output the concrete proposed change (e.g. "add
        collabora WOPI healthcheck + start_period", "add pg_dump backup hook").
      - **CI-SERVER bug** → needs a **cc-ci PR** (harness/test/infra); output the proposed change.
      - **TEST out-of-date** → the recipe legitimately changed; the cc-ci test needs updating; output
        the update.
      - **FLAKY / infra** → distinguish from a real failure (e.g. re-run) so a flake isn't
        misclassified as a recipe/CI bug.
- [ ] **R6 — Structured report; PROPOSE not auto-merge.** Output a structured report (per recipe
      status; per failure: root cause + classification + proposed PR target + change sketch), or the
      all-green summary. It **proposes** PRs — it does **NOT** auto-create or merge them (the operator
      merges recipe PRs only after cc-ci verifies them green, per the recipe-PR rule). It MAY offer to
      hand a proposed recipe PR to the `recipe-create-pr` skill as an explicit, opt-in follow-up.
- [ ] **R7 — Documented as distinct from deterministic CI.** `docs/` makes clear the default
      `!testme`/nightly path is deterministic (no AI); `/ci-test-review` is the on-demand AI review
      layer. A maintainer who wants zero AI never invokes it.
- [ ] **R8 — Adversary-verified behavior.** Proven: reports ALL-GREEN correctly when green; on a
      **seeded recipe bug** classifies → recipe-PR with a sane proposed change; on a **seeded harness
      bug** classifies → CI-PR; the freshness check flags a deliberately stale-pinned recipe; a flake
      isn't misreported as a real bug.

## 2. Method / notes
- **Lean on the harness + the Adversary's existing diagnostic methodology.** The classification
  (recipe vs CI vs test-stale) is exactly what the loops already do per finding — the skill
  productizes that reasoning, with the harness providing the deterministic evidence.
- **Real abra path** throughout (the harness it drives already uses real abra).
- The skill's value is turning "N recipes, here's the raw pass/fail" into "here's what's actually
  wrong and the specific PR that fixes it (recipe vs CI), or all-clear."

## 3. Guardrails
- **Deterministic execution stays AI-free** — AI only diagnoses/classifies/proposes; it never
  decides pass/fail.
- **Propose, don't auto-merge** — recipe PRs are operator-merged only after cc-ci verifies them
  (the standing recipe-PR rule).
- **Don't weaken any test** to make a review "pass."
- **Bounded** — one review skill; not a general agent. It runs the suite, diagnoses, reports.

## 4. Open decisions (log in machine-docs/DECISIONS.md)
- Cold vs `--quick` default for the review run (lean cold = authoritative; offer `--quick` fast mode).
- Report format (markdown summary + per-recipe table + per-failure diagnosis block).
- How "latest published version" + "test staleness" are determined per recipe.
- Whether/how the skill optionally invokes `recipe-create-pr` for a proposed recipe PR (opt-in).