Files
cc-ci-orchestrator/.claude/skills/ci-test-review/SKILL.md
T
notplantsandClaude Opus 4.8 2530845e50 orchestrator: add /ci-test-review skill (in THIS repo) + drop Phase 3r from loops queue
The on-demand AI review layer is now an orchestration-repo skill built directly
by the orchestrator, NOT a loops phase in the cc-ci product repo:

- .claude/skills/ci-test-review/{SKILL.md,run-all-recipes.sh}: runs the real
  cc-ci harness across all enrolled recipes (deterministic, AI-free execution),
  then AI diagnoses each failure and classifies it as needing a recipe PR / a
  CI-server PR / a stale-test update — or reports "ALL PASSED, recipes + tests
  up to date". Proposes PRs; never decides pass/fail; never auto-merges.
- .gitignore: track .claude/skills/ (shareable) while still ignoring local
  claude session state (locks, history) under .claude/.
- launch.sh: remove Phase 3r from PHASES_SPEC; loops sequence back to
  1c 1b 1d 1e 2w 2pc 2 2b 3 4. Deleted plan-phase3r (superseded by the skill).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-05-29 16:57:26 +01:00

4.9 KiB

name, description
name description
ci-test-review On-demand AI review of the cc-ci server — runs the deterministic recipe test suite across ALL enrolled recipes, and for any failure diagnoses the root cause and classifies it as needing a RECIPE PR (with the proposed change), a CI-SERVER PR, or a stale-test update; or reports "all passed, recipes + tests up to date". Use when the operator wants a full health check + actionable diagnosis of the recipe CI, beyond the deterministic pass/fail of normal !testme runs. Invoke as /ci-test-review.

ci-test-review

An on-demand, AI-driven review of the cc-ci recipe CI server. cc-ci's primary use is deterministic testing (normal !testme / nightly — no AI). This skill adds a review layer: run the full suite across all recipes, then AI-diagnose any failures and say exactly what fixes them — a recipe PR, a CI-server PR, or a test update — or confirm everything is green and current.

Hard boundary: the test execution is deterministic (the cc-ci harness). AI is used ONLY for diagnosis / classification / proposing the PR — never to decide pass/fail.

Access (this skill runs from the orchestration repo, drives cc-ci over the tailnet)

  • cc-ci is reachable via ssh cc-ci (root, through the userspace-tailscaled SOCKS proxy on 127.0.0.1:1055). If ssh cc-ci fails, restart the proxy (systemctl restart cc-ci-tailscaled) per the orchestrator's access notes, then retry.
  • The cc-ci repo + harness live at /root/cc-ci on the cc-ci VM; the runner is cc-ci-run runner/run_recipe_ci.py with RECIPE=<name> (+ optional STAGES=...).

Procedure

1. Run the deterministic sweep (no AI)

Run the helper, which enumerates enrolled recipes and runs the full suite per recipe on cc-ci, emitting one structured result line per recipe (per-tier pass/fail/skip + the tested vs latest published version):

bash "$(dirname "$0")/run-all-recipes.sh"            # or: .claude/skills/ci-test-review/run-all-recipes.sh

It writes a results table to stdout and a machine-readable summary to /srv/cc-ci/.cc-ci-logs/ci-test-review-<runid>.json. Do NOT hand-judge pass/fail — trust the harness's per-tier results.

2. If everything is green AND current → report all-clear

If every recipe's tiers are pass/skip(N/A) (no fail) and every recipe was tested at its latest published version with no stale test flagged, output:

ALL PASSED — recipes up to date, tests up to date. <N recipes, versions listed>

Stop here. Nothing to fix.

3. For each failure → diagnose root cause + CLASSIFY (this is the AI part)

For every fail (after confirming it's not a flake — see below), read the run log, the recipe's compose/code (on cc-ci ~/.abra/recipes/<recipe>), and the harness, then determine the root cause and classify it as exactly one of:

  • RECIPE bug → recipe PR. The recipe itself is broken/fragile (bad healthcheck, missing backup hook, startup race, etc.). Output the concrete proposed change (e.g. "add a collabora WOPI healthcheck + start_period", "add a pg_dump backup hook"). This is a PR to the recipe (recipe-maintainer); it is "working" only once cc-ci verifies it green, then the operator merges.
  • CI-SERVER bug → cc-ci PR. The harness/test/infra is wrong (a test asserts the wrong thing, a readiness gate is off, an infra config bug). Output the proposed change to the cc-ci repo.
  • TEST out-of-date → cc-ci test update. The recipe legitimately changed upstream and the cc-ci test/overlay needs updating to match. Output the update.
  • FLAKY / infra-transient. Distinguish from a real failure: re-run the recipe once or twice; if it then passes, classify as flaky (note it; suggest a robustness fix if there's a pattern) — do NOT report a flake as a recipe/CI bug.

4. Freshness (part of "up to date")

For each recipe, compare the tested version to the latest published recipe version (the sweep records both). Flag any recipe tested at a stale pin, and any cc-ci test that lags a recipe change — "all passed" must mean "passes against current upstream", not against an old pin.

5. Output a structured report

  • A per-recipe status table (recipe | tiers | version tested / latest | verdict).
  • For each failure: root cause → classification (recipe PR / CI PR / test-update / flaky) → proposed change (concrete enough to act on).
  • Or the all-clear summary from step 2.

Guardrails

  • Deterministic execution stays AI-free — only diagnose; never override the harness's pass/fail.
  • Propose, don't auto-merge. This skill outputs recommendations + proposed PRs. It does NOT create or merge PRs. (A recipe PR is operator-merged only after cc-ci verifies it green.) If asked, it may hand a specific proposed recipe PR to the recipe-create-pr skill as an explicit follow-up.
  • Don't weaken any test to make the review "pass."
  • Flake ≠ bug — re-run before classifying a failure as a real recipe/CI defect.