recipe-maintainers/cc-ci-orchestrator
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
e89da2d842bd9ea918adc4e0219955418d2e5a6b
cc-ci-orchestrator/.claude/skills/ci-test-review/SKILL.md
T
autonomic-bot 6c91373357 skills: point every test-editing path at tests/STYLE.md 
/recipe-upgrade --with-tests, /ci-test-review and /cc-ci-tests-update all author
test changes, and all three had only 'never weaken a test' as guidance. That did
not stop the plausible failure: the fixture INSERTed rows into the app's database,
which was correct for v2 and silently wrong for v3, where a site must belong to a
team. Events were acked 202 and discarded; the recipe sat RED for six weeks.

The rule that would have prevented it — set state up through the app's own
interface, not its database — now lives in tests/STYLE.md in the cc-ci repo, and
each of the three paths is told to read it before editing a test.
2026-08-11 14:41:10 +00:00

8.6 KiB
Raw Blame History

name, description
name description
ci-test-review On-demand AI review + auto-fix of the cc-ci server. Runs the deterministic recipe test suite across ALL enrolled recipes; for any failure it diagnoses the root cause, classifies it as a RECIPE bug or a CI-SERVER bug, AUTHORS the fix, OPENS a PR (recipe repo or cc-ci repo), and VERIFIES that PR green on the CI server (deploy + full suite against the PR head, dogfood) — but NEVER merges (the operator merges). If everything is green and current it reports "all passed, recipes + tests up to date". Use for a full health check that ends in verified, ready-to-merge fix PRs. Invoke as /ci-test-review.

ci-test-review

An on-demand, AI-driven review + auto-fix layer over the cc-ci recipe CI server. cc-ci's primary use is deterministic testing (normal !testme / nightly — no AI). This skill runs the full suite across all recipes, then for any failure: diagnoses the root cause, classifies it (recipe vs CI-server), authors the fix, opens a PR, and verifies that PR green on the CI server — the same deploy-and-test dogfood the build loops use. It leaves a verified, ready-to-merge PR; it does NOT merge. If everything's green and current it reports "all passed".

This skill drives its own CI server (cc-ci) the way the loops do: it can push branches, deploy recipes, and run the harness there to verify a fix end-to-end.

Hard boundaries:

  • The test execution and PR verification are deterministic (the cc-ci harness decides pass/fail). AI never decides pass/fail — it only diagnoses, authors the fix, and drives the flow.
  • Create + verify, never merge. The operator reviews the cc-ci-green PR and merges.
  • Never weaken a test to make a PR go green.

Access (this skill runs from the orchestration repo, drives cc-ci over the tailnet)

  • cc-ci is reachable via ssh cc-ci (root, through the userspace-tailscaled SOCKS proxy on 127.0.0.1:1055). If ssh cc-ci fails, restart the proxy (systemctl restart cc-ci-tailscaled) per the orchestrator's access notes, then retry.
  • The cc-ci repo + harness live at /root/cc-ci on the cc-ci VM; the runner is cc-ci-run runner/run_recipe_ci.py with RECIPE=<name> (+ optional STAGES=...).

Procedure

1. Run the deterministic sweep (no AI)

Run the helper, which enumerates enrolled recipes and runs the full suite per recipe on cc-ci, emitting one structured result line per recipe (per-tier pass/fail/skip + the tested vs latest published version):

bash "$(dirname "$0")/run-all-recipes.sh"            # or: .claude/skills/ci-test-review/run-all-recipes.sh

It writes a results table to stdout and a machine-readable summary to /srv/cc-ci/.cc-ci-logs/ci-test-review-<runid>.json. Do NOT hand-judge pass/fail — trust the harness's per-tier results.

2. If everything is green AND current → report all-clear

If every recipe's tiers are pass/skip(N/A) (no fail) and every recipe was tested at its latest published version with no stale test flagged, output:

ALL PASSED — recipes up to date, tests up to date. <N recipes, versions listed>

Stop here. Nothing to fix.

3. For each failure → diagnose root cause + CLASSIFY (AI)

For every fail (after confirming it's not a flake — see below), read the run log, the recipe's compose/code (on cc-ci ~/.abra/recipes/<recipe>), and the harness, then determine the root cause and classify it as exactly one of:

  • RECIPE bug → recipe-side fix. The recipe itself is broken/fragile (bad healthcheck, missing backup hook, startup race, etc.). The fix is a change to the recipe (e.g. "add a collabora WOPI healthcheck + start_period", "add a pg_dump backup hook").
  • CI-SERVER bug → cc-ci-side fix. The harness/test/infra is wrong (a test asserts the wrong thing, a readiness gate is off, an infra config bug). The fix is a change to the cc-ci repo.
  • TEST out-of-date → cc-ci-side fix. The recipe legitimately changed upstream and the cc-ci test/overlay needs updating to match.
  • FLAKY / infra-transient. Distinguish from a real failure: re-run the recipe once or twice; if it then passes, classify as flaky (note it; suggest a robustness fix if there's a pattern) — do NOT author a recipe/CI "fix" for a flake.

Freshness is part of this: compare each recipe's tested version to the latest published version (the sweep records both). A recipe tested at a stale pin, or a cc-ci test lagging an upstream recipe change, is itself a finding — "all passed" must mean "passes against current upstream".

4. Author the fix + OPEN a PR (AI authors; never merge)

For each real (non-flaky) finding, write the actual fix and open a PR. Never merge.

  • Recipe-side fix → recipe PR. Branch the recipe, apply the fix, and open the PR via the recipe-create-pr skill (/srv/recipe-maintainer/.opencode/skills/recipe-create-pr/SKILL.md) — it handles the mirror to git.autonomic.zone/recipe-maintainers/<recipe> (upstream git.coopcloud.tech). Keep the change bounded to the diagnosed root cause; don't rewrite the recipe.
  • Before editing any test, read tests/STYLE.md in the cc-ci repo. It encodes the rules a test change must satisfy — set state up through the app's interface rather than its database, gate on version instead of branching, correct the fixture/wait but NEVER the assertion, and diagnose from the app's own telemetry before concluding a test is stale.
  • CI-server-side fix → cc-ci PR. Branch the cc-ci product repo (recipe-maintainers/cc-ci), apply the fix, and open the PR via the Gitea API (use the GITEA_* creds from /srv/cc-ci/.testenv). Single-writer discipline: work on a dedicated branch in a SEPARATE clone — never push main, never touch the build loops' working clones (/cc-ci, /cc-ci-adv) or their in-flight state.

5. VERIFY each PR on the CI server (deterministic; still never merge)

A PR is only "working" once cc-ci verifies it green (operator rule) — dogfood the CI that found the bug. Verification is deterministic (the harness), not an AI judgement.

  • Recipe PR: run the full suite COLD against the PR head on cc-ci — the same path !testme uses — via the helper: 
    RECIPE=<recipe> REF=<pr-head-branch-or-sha> .claude/skills/ci-test-review/verify-pr.sh
    Green ⇔ the harness exits 0. General bar = one cold green. Use REPEAT=3 (repeated-green) only for a recipe already known to be FLAKY (e.g. lasuite-drive) as a flakiness proof — this is NOT the general standard.
  • CI-server PR: check the branch out in a clone on cc-ci, rebuild if the change requires it (nixos-rebuild / flake), then re-run the previously-failing recipe(s) through the harness plus a small regression sample of unrelated recipes. Green ⇔ the fix passes AND nothing regressed.
  • If verification is RED: iterate the fix on the same branch (bounded — ≤3 attempts) and re-verify. If still red, leave the PR open and report it as "opened, NOT yet green — needs work" with the failing evidence. Never abandon silently; never weaken a test to force green.

6. Output a structured report

  • A per-recipe status table (recipe | tiers | version tested / latest | verdict).
  • For each finding: root cause → classification (recipe / CI-server) → PR link → verification verdict (GREEN — cold-verified on cc-ci, ready for operator merge / RED — opened, needs work).
  • Or the all-clear summary from step 2.
  • Always end with the explicit reminder that nothing was merged — the PRs await operator review.

Guardrails

  • Deterministic execution + verification stay AI-free — the harness decides pass/fail; AI only diagnoses, authors the fix, and drives the flow.
  • Create + verify, NEVER merge. Leave a cc-ci-green, ready-to-merge PR; the operator merges (recipe PRs especially are operator-merged only after cc-ci verifies them green).
  • Don't weaken any test to make a PR go green — the fix must make the recipe/CI genuinely correct.
  • Flake ≠ bug — re-run before authoring any fix.
  • Real abra path throughout (no docker-level bypass).
  • Bounded fixes — each PR targets one diagnosed root cause; not a rewrite.
  • Coordination / single-writer: while the build loops are actively developing cc-ci, the server and the recipe-maintainers/cc-ci repo are in use. Always use a dedicated branch + separate clone; never push main or disturb the loops' clones. Recipe deploys are stateful on the shared Swarm, so run verification when you have effectively-exclusive use of the host (or serialize), and tear down whatever you deploy.
Reference in New Issue View Git Blame Copy Permalink