--- name: ci-test-review description: On-demand AI review of the cc-ci server — runs the deterministic recipe test suite across ALL enrolled recipes, and for any failure diagnoses the root cause and classifies it as needing a RECIPE PR (with the proposed change), a CI-SERVER PR, or a stale-test update; or reports "all passed, recipes + tests up to date". Use when the operator wants a full health check + actionable diagnosis of the recipe CI, beyond the deterministic pass/fail of normal !testme runs. Invoke as /ci-test-review. --- # ci-test-review An **on-demand, AI-driven review** of the cc-ci recipe CI server. cc-ci's primary use is **deterministic** testing (normal `!testme` / nightly — no AI). This skill adds a review layer: run the full suite across **all** recipes, then **AI-diagnose** any failures and say exactly what fixes them — a recipe PR, a CI-server PR, or a test update — or confirm everything is green and current. **Hard boundary:** the test *execution* is deterministic (the cc-ci harness). **AI is used ONLY for diagnosis / classification / proposing the PR — never to decide pass/fail.** ## Access (this skill runs from the orchestration repo, drives cc-ci over the tailnet) - cc-ci is reachable via `ssh cc-ci` (root, through the userspace-tailscaled SOCKS proxy on 127.0.0.1:1055). If `ssh cc-ci` fails, restart the proxy (`systemctl restart cc-ci-tailscaled`) per the orchestrator's access notes, then retry. - The cc-ci repo + harness live at `/root/cc-ci` on the cc-ci VM; the runner is `cc-ci-run runner/run_recipe_ci.py` with `RECIPE=` (+ optional `STAGES=...`). ## Procedure ### 1. Run the deterministic sweep (no AI) Run the helper, which enumerates enrolled recipes and runs the **full suite per recipe** on cc-ci, emitting one structured result line per recipe (per-tier pass/fail/skip + the tested vs latest published version): ``` bash "$(dirname "$0")/run-all-recipes.sh" # or: .claude/skills/ci-test-review/run-all-recipes.sh ``` It writes a results table to stdout and a machine-readable summary to `/srv/cc-ci/.cc-ci-logs/ci-test-review-.json`. Do NOT hand-judge pass/fail — trust the harness's per-tier results. ### 2. If everything is green AND current → report all-clear If every recipe's tiers are `pass`/`skip(N/A)` (no `fail`) **and** every recipe was tested at its **latest published version** with no stale test flagged, output: > **ALL PASSED — recipes up to date, tests up to date.** Stop here. Nothing to fix. ### 3. For each failure → diagnose root cause + CLASSIFY (this is the AI part) For every `fail` (after confirming it's not a flake — see below), read the run log, the recipe's compose/code (on cc-ci `~/.abra/recipes/`), and the harness, then determine the **root cause** and classify it as **exactly one** of: - **RECIPE bug → recipe PR.** The recipe itself is broken/fragile (bad healthcheck, missing backup hook, startup race, etc.). Output the **concrete proposed change** (e.g. "add a collabora WOPI healthcheck + `start_period`", "add a `pg_dump` backup hook"). This is a PR to the recipe (recipe-maintainer); it is "working" only once cc-ci verifies it green, then the operator merges. - **CI-SERVER bug → cc-ci PR.** The harness/test/infra is wrong (a test asserts the wrong thing, a readiness gate is off, an infra config bug). Output the proposed change to the cc-ci repo. - **TEST out-of-date → cc-ci test update.** The recipe legitimately changed upstream and the cc-ci test/overlay needs updating to match. Output the update. - **FLAKY / infra-transient.** Distinguish from a real failure: **re-run the recipe once or twice**; if it then passes, classify as flaky (note it; suggest a robustness fix if there's a pattern) — do NOT report a flake as a recipe/CI bug. ### 4. Freshness (part of "up to date") For each recipe, compare the **tested version** to the **latest published** recipe version (the sweep records both). Flag any recipe tested at a stale pin, and any cc-ci test that lags a recipe change — "all passed" must mean "passes against current upstream", not against an old pin. ### 5. Output a structured report - A per-recipe **status table** (recipe | tiers | version tested / latest | verdict). - For each failure: **root cause → classification (recipe PR / CI PR / test-update / flaky) → proposed change** (concrete enough to act on). - Or the all-clear summary from step 2. ## Guardrails - **Deterministic execution stays AI-free** — only diagnose; never override the harness's pass/fail. - **Propose, don't auto-merge.** This skill outputs recommendations + proposed PRs. It does NOT create or merge PRs. (A recipe PR is operator-merged only after cc-ci verifies it green.) If asked, it may hand a specific proposed recipe PR to the `recipe-create-pr` skill as an explicit follow-up. - **Don't weaken any test** to make the review "pass." - **Flake ≠ bug** — re-run before classifying a failure as a real recipe/CI defect.