cc-ci-orchestrator/.opencode/skills/cc-ci-tests-update/SKILL.md

name, description

name

description

cc-ci-tests-update

Sweep all maintained recipes for stale cc-ci tests and update the stale ones so they pass again, verified via the standard PR + !testme -> CI flow. Runs the deterministic test suite across every enrolled recipe, identifies failures caused by a genuinely-stale test (the recipe legitimately changed upstream and the cc-ci test/overlay still asserts the old behavior), authors the minimal test update, opens a cc-ci test PR per stale recipe, and verifies each by running !testme on the corresponding recipe upgrade PR with the test change applied (or a direct harness run with the cc-ci branch checked out). NEVER weakens a test to force green; never merges. The fleet-wide stale-test cleanup that the weekly /upgrade-all deliberately defers (it comments-only on stale tests; this skill closes the loop). Invoke as /cc-ci-tests-update.

cc-ci-tests-update

The fleet-wide stale-test cleanup for cc-ci. The weekly /upgrade-all cron deliberately does not auto-edit cc-ci tests — it runs in DEFAULT mode and, where a recipe's existing test is genuinely stale against a new upstream version, it leaves an explanatory comment on the recipe PR and stops (operator decides). Over time those commented stale tests accumulate. This skill closes that loop: it sweeps every maintained recipe, finds the ones whose cc-ci test is stale, authors the minimal test update, opens a cc-ci test PR per stale recipe, and verifies each via the standard !testme → CI flow (or a direct harness run with the test branch applied). It never weakens a test and never merges.

Relationship to the other skills. This is the fleet-wide version of the per-recipe --with-tests path in /recipe-upgrade (step 5b), and the "TEST out-of-date" classification branch of /ci-test-review. It reuses both skills' discipline + helpers. The difference is scope: those skills react to a failure they happened to hit; this one proactively sweeps the whole fleet for stale tests and fixes them in one pass.

What this skill does NOT do

• Does NOT bump recipe image tags — that's /upgrade-all / /recipe-upgrade. This skill fixes the tests, not the recipes. (It may discover a stale test by re-running a recipe upgrade PR's !testme, but the recipe bump itself is owned by /upgrade-all.)
• Does NOT touch the cc-ci server host / flake — that's /cc-ci-server-update.
• Does NOT fix real upgrade regressions — if a recipe's !testme is red because the upgrade is broken (not because the test is stale), that's a /recipe-upgrade (step 5a) or /ci-test-review job. This skill only touches a test when the upgrade is correct and the test is wrong.
• Does NOT merge any PR — operator merges after review.

Preconditions / access

• ssh cc-ci works (root). Same access notes as /recipe-upgrade / /ci-test-review.
• /srv/cc-ci/.testenv has the GITEA_* creds (for opening cc-ci test PRs + posting !testme).
• The shared Swarm is quiescent enough for test runs — each stale-test verification deploys a recipe, so don't run this concurrent with a /upgrade-all or active build loops. Serialize.
• abra over ssh needs the pseudo-TTY wrapper (see the box in /recipe-upgrade): every abra call is ssh cc-ci 'script -qec "abra <args> -n" /dev/null'.

Procedure

1. Build the stale-test candidate list (sweep — deterministic, no AI judgement yet)

The candidates are recipes whose cc-ci test is potentially stale. Two sources, merge both:

(a) Carry-over from /upgrade-all — the commented stale tests. Each weekly run's report has a "PRs where a test looks stale" section (recipes where !testme went RED on a stale test and got a PR comment). Read the latest report + a few weeks back:

grep -A2 "test.*stale\|stale.*test\|SUCCESS-PENDING-TESTS" /srv/cc-ci/.cc-ci-logs/upgrades/upgrade-all-*.md | tail -40

Each line names a recipe + the stale test + the PR URL. These are known stale (already diagnosed by the upgrader) — they go straight to the fix list.

(b) Fresh sweep — re-run the test suite across all maintained recipes. Use the /ci-test-review deterministic sweep helper to find failures that aren't already on the carry-over list:

bash /srv/cc-ci-orch/.claude/skills/ci-test-review/run-all-recipes.sh

It runs the full harness per recipe on cc-ci and writes a JSON summary to /srv/cc-ci/.cc-ci-logs/ci-test-review-<runid>.json. Parse it for fail verdicts.

Concurrency note: a full sweep is ~20 recipes × full suite — it takes a while and uses the shared Swarm. Run it when the host is otherwise idle. If you only want the carry-over stale tests (skip the fresh sweep), pass RECIPES="recipe1 recipe2" to the helper to limit it.

(c) Merge + classify. Combine (a) and (b) into CANDIDATES. For each candidate, classify the failure (AI — this is the ci-test-review step-3 diagnosis):

• RECIPE bug (the upgrade is broken) → NOT this skill. Hand off to /recipe-upgrade (5a) or note it for the operator. Don't touch the test.
• CI-SERVER bug (harness/infra wrong) → NOT this skill (that's a /ci-test-review cc-ci-side fix, not a test-update). Note it.
• TEST out-of-date (the recipe legitimately changed upstream and the cc-ci test/overlay asserts old behavior) → this skill. Add to STALE_TESTS with: recipe, the failing test name, what changed upstream, what the test currently asserts.
• FLAKY → re-run once or twice; if it passes, drop it (not stale, just flaky).

2. For each stale test — author the minimal test update (AI; never weaken)

Work on one recipe at a time (serialize — each verification deploys a recipe on the shared Swarm). For each STALE_TESTS entry:

(a) Set up a dedicated cc-ci clone on a branch (single-writer — never push main, never touch the build loops' clones):

git clone ssh://git@git.autonomic.zone:2222/recipe-maintainers/cc-ci.git /home/loops/work/cc-ci-tests-<recipe>
cd /home/loops/work/cc-ci-tests-<recipe>
git fetch origin && git checkout main && git reset --hard origin/main
git checkout -b test/<recipe>-stale-test-$(date -u +%Y%m%d)

(Or reuse /home/loops/work/cc-ci-fix on a new branch if it's clean — the rule is a dedicated branch off main, in a clone you're not racing the loops on.)

(b) Read the failing test + the upstream change. The test lives in tests/<recipe>/ (custom tests, overlays, fixtures). Read:

• the failing test (the sweep / carry-over names it),
• the recipe's current compose.yml + upstream changelog (use the per-recipe registry cc-ci-plan/upstream/<recipe>.md for the release-notes URLs — the /upgrade-all diagnosis already identified what changed),
• what the test currently asserts vs what the new version does.

(c) Author the minimal update. Change the test/overlay to assert the new, correct behavior — not to weaken it. Examples of a legitimate stale-test fix:

• a config key was renamed upstream → update the overlay to the new key.
• a default flipped (e.g. allow_uncategorized_topics false→true in Discourse 3.5) → update the assertion to the new default, OR make the test explicitly set the old value (preferred — the test then asserts the behavior it cares about, not the default).
• an auth method was removed (e.g. Bearer-auth in impress v5.4.0) → drop that assertion, or replace it with the new auth method's assertion (preferred if the new method is the successor).
• a readiness endpoint changed → update the readiness gate to the new endpoint.

What is NOT allowed: lowering an assertion's strictness to make a red go green, deleting a test that still applies, skipping a tier that still applies, or changing a pass threshold. The fix must make the test correct for the new version, not lenient. If you can't fix it without weakening, stop and report FAILED — test cannot be updated without weakening; needs operator decision (leave the recipe's stale-test PR comment as-is).

(d) Lint the test change (cc-ci has a lint gate):

ruff check tests/<recipe>/ && ruff format --check tests/<recipe>/

Fix any lint errors you introduce (don't commit red lint).

3. Open the cc-ci test PR (never merge)

git add tests/<recipe>/
git commit -m "test(<recipe>): update stale test for <upstream-change>

<test-name> asserted <old-behavior>; <recipe> <new-version> now <new-behavior>.
Updated to assert the new behavior. No assertion weakened.

Stale-test fix for recipe PR <url> (carry-over from /upgrade-all <date>)."
git push -u origin HEAD

Open the PR via the helper:

set -a; . /srv/cc-ci/.testenv; set +a
TITLE="test(<recipe>): update stale test for <upstream-change>" \
BODY_FILE=/tmp/cc-ci-tests-<recipe>-pr-body.md \
bash /srv/cc-ci-orch/.claude/skills/ci-test-review/open-cc-ci-pr.sh

The PR body must: name the stale test, explain what changed upstream, state that no assertion was weakened, link the dependent recipe upgrade PR (the test change only goes green once the recipe upgrade is also applied — they're a pair), and say "NOT merged — for operator review."

Capture the TEST_PR_URL.

4. Verify the updated test passes (deterministic; the standard PR + !testme flow)

A test-update PR is only "working" once cc-ci verifies it green with the new test applied. There are two verification paths — use whichever applies:

(a) If the stale test was carry-over from an open recipe upgrade PR — verify via the paired !testme flow (the /recipe-upgrade --with-tests step-5b path). The recipe upgrade PR's !testme uses the deployed/main cc-ci tests, so it can't see your unmerged test branch — you need to run the harness with your cc-ci test branch checked out on cc-ci:

# 1. On cc-ci, check out your test branch in a deploy clone (NOT /root/builder-clone — use a
#    separate checkout so the running server is unaffected):
ssh cc-ci 'rm -rf /root/cc-ci-test-verify && git clone --branch test/<recipe>-stale-test-<date> \
  ssh://git@git.autonomic.zone:2222/recipe-maintainers/cc-ci.git /root/cc-ci-test-verify && \
  cd /root/cc-ci-test-verify && git submodule update --init secrets 2>/dev/null || true'
# 2. Run the harness against the recipe upgrade PR head, with your test branch as the cc-ci checkout:
RECIPE=<recipe> REF=<recipe-upgrade-pr-head-branch> REMOTE_ROOT=/root/cc-ci-test-verify \
  bash /srv/cc-ci-orch/.claude/skills/ci-test-review/verify-pr.sh

Green ⇔ the recipe upgrade passes under your corrected test. This is the paired verification (the test change + the recipe upgrade together go green).

(b) If the stale test is against a recipe already at its current upstream (no upgrade PR — the test is just stale against current main) — verify the test branch directly:

RECIPE=<recipe> REMOTE_ROOT=/root/cc-ci-test-verify \
  bash /srv/cc-ci-orch/.claude/skills/ci-test-review/verify-pr.sh

(No REF — runs against the current recipe main, with your updated tests.)

(c) Cleanup the verify clone on cc-ci (mandatory — shared host):

ssh cc-ci 'rm -rf /root/cc-ci-test-verify'

If verification is RED: iterate the test fix on the same branch (bounded — ≤3 attempts) and re-verify. If still red after 3, leave the test PR open and report FAILED — test update not green after 3 verify runs with the failing evidence. Don't weaken the test to force green.

Re-verify with a real !testme on the recipe PR too? If the recipe upgrade PR is still open, posting a fresh !testme on it after the test PR merges would re-run with the merged tests — but you can't merge the test PR (operator does). So the harness-branch verification in (a) is the authoritative pre-merge proof. Leave a comment on the recipe upgrade PR noting the test PR is open and verified-paired, so the operator knows merging both → green.

5. Tear down any dev deploys (defence-in-depth)

If your verification path (or diagnosis) deployed a dev-<recipe> stack on cc-ci, tear it down:

ssh cc-ci 'D=dev-<recipe>.ci.commoninternet.net; script -qec "abra app undeploy $D -n" /dev/null 2>&1 | tail -2; \
  script -qec "abra app rm $D -n --no-input" /dev/null 2>&1 | tail -2'
ssh cc-ci 'docker stack ls --format "{{.Name}}" | grep -c "^dev-<recipe>"'   # should be 0

(The /upgrade-all sweep would reap it next run anyway, but clean up yourself.)

6. Report — one line per recipe + summary

Write /srv/cc-ci/.cc-ci-logs/tests-update-<YYYY-MM-DD>.md and print, leading with the PR list:

# cc-ci Stale-Test Update — <YYYY-MM-DD>
## Summary
- Swept: N recipes · Stale tests found: N · Test PRs opened + verified: N · Failed: N · Not-stale (dropped): N
## Test PRs to review (NOT merged)
| recipe | stale test | upstream change | test PR | verified |
|--------|-----------|-----------------|---------|----------|
| <recipe> | <test-name> | <what changed> | <url> | GREEN (paired with recipe PR <url>) |
## Failed (needs work)
- <recipe> — <test-name>: <reason> (test PR <url> left open, red)
## Not-stale (dropped — was flaky / real recipe bug / CI-server bug)
- <recipe> — <test-name>: <classification> (handed off / noted)

End with the report path and a reminder that nothing was merged — the test PRs (and their paired recipe upgrade PRs) await operator review.

Guardrails

• Never weaken a test to turn a red green. The fix makes the test correct for the new upstream behavior; if that's not possible without weakening, stop and report.
• Create + verify, NEVER merge. Each test PR (and any paired recipe upgrade PR) is operator-merged after review.
• Single-writer: dedicated branch in a separate clone; never push main; never touch the build loops' /cc-ci /cc-ci-adv clones or /root/builder-clone. Use /root/cc-ci-test-verify on the host and rm -rf it after.
• Stale ≠ broken. Only update a test when the upgrade/recipe is correct and the test is wrong (asserts old behavior). A real upgrade regression is a /recipe-upgrade (5a) job; a CI-server bug is a /ci-test-review job. This skill fixes tests, not recipes or harness.
• Bounded fixes — minimal update to the failing assertion/overlay, not a test rewrite.
• Bounded retries — ≤3 verify runs per test PR; then leave it open + red with evidence.
• Flake ≠ stale — re-run before classifying as stale; don't author a fix for a flake.
• Serialize on the shared Swarm — one recipe's verify deploy at a time; tear down before the next. Don't run concurrent with /upgrade-all or active build loops.
• Real abra path throughout (no docker-level bypass); pseudo-TTY wrap every abra call.