cc-ci-orchestrator/.claude/skills/ci-dev-workflow/SKILL.md

name, description

name

description

ci-dev-workflow

End-to-end workflow for developing a feature or fix in the cc-ci CI server (the harness, the level/results/card system, or a recipe's tests) from the orchestrator. Covers exploring the harness, branching a local clone, implementing with unit + live verification, opening a PR, getting an independent adversary verdict, merging, and deploying to the running CI server. Use whenever you are changing cc-ci server code (runner/harness/**, tests/**) rather than just running recipe upgrades. Invoke as /ci-dev-workflow.

ci-dev-workflow

How to safely build and ship a change to the cc-ci CI server from the orchestrator. This is the flow used to land the "intentional skips + 4-rung level ladder" feature. It never pushes main and never merges without an independent adversary PASS.

The two repos / two hosts

• Orchestrator (where you are): /srv/cc-ci-orch and /srv/cc-ci are both checkouts of the cc-ci-orchestrator repo (plans, skills, launchers). Edit /srv/cc-ci-orch; push to git.autonomic.zone/recipe-maintainers/cc-ci-orchestrator.
• cc-ci CI server: reached via ssh cc-ci. The CI server's own source is the cc-ci repo (https://autonomic-bot:<token>@git.autonomic.zone/recipe-maintainers/cc-ci.git — the token is in the /root/builder-clone remote and in /srv/cc-ci/.testenv as GITEA_*). The harness lives in runner/harness/**, the orchestrator entry is runner/run_recipe_ci.py, tests in tests/**.

Key constraint: there is NO python3/pytest on the orchestrator host. Run all Python — unit tests, card/badge rendering, live harness runs — ON the cc-ci host via ssh cc-ci, using the cc-ci-run wrapper (it provides python + pytest + playwright/chromium + the harness env).

Harness orientation (read before editing)

• runner/harness/level.py — PURE level ladder. Cardinal invariant: presentation must never look greener than the tests. The ladder is the four ESSENTIAL rungs install · upgrade · backup_restore · functional (top = L4); a gap (FAIL or N/A) caps the climb. integration/recipe-local are optional and NOT leveled. Keep this module pure + unit-tested.
• runner/harness/results.py — builds results.json (derive_rungs, skips, build_results). skips splits N/A rungs into intentional (declared in recipe_meta.EXPECTED_NA={rung:reason}) vs unintentional. Results assembly is best-effort (R7): a failure here NEVER changes the run verdict.
• runner/harness/card.py — summary card (HTML→PNG via headless chromium) + the level badge SVG.
• runner/run_recipe_ci.py — the orchestrator: deploys once, runs the tiers (install,upgrade,backup,restore,custom), computes the verdict. The run VERDICT (SSO/F2-11, teardown, leak scan) is separate from the cosmetic level/card — never weaken verdict enforcement.
• A recipe's tests live in tests/<recipe>/: recipe_meta.py (timeouts, EXPECTED_NA, BACKUP_CAPABLE, DEPS, …), functional/ + playwright/ (custom tier), install_steps.sh (pre-deploy hook), ops.py (pre-op seed hooks). The generic tiers are tests/_generic/.

Steps

1. Explore first

Read the modules you'll touch + their unit tests (tests/unit/test_*.py) and any recipe under tests/<recipe>/. Understand the verdict-vs-cosmetic split and the cardinal "never inflate" rule. For a recipe-behaviour question, also look at the recipe on the host: ssh cc-ci 'cat ~/.abra/recipes/<recipe>/compose*.yml'.

2. Branch a local clone of cc-ci (on the orchestrator)

So you can edit with real tooling (Read/Edit/Write) instead of ssh+heredocs:

cd /tmp && git clone "https://autonomic-bot:<token>@git.autonomic.zone/recipe-maintainers/cc-ci.git" cc-ci-feature
cd cc-ci-feature && git checkout -b feat/<slug>
git config user.name autonomic-bot && git config user.email autonomic-bot@git.autonomic.zone

3. Implement

Edit the harness + tests in the local clone. Match surrounding style. For pure logic (level/results), add focused unit tests in tests/unit/. For recipe behaviour, add a tests/<recipe>/functional/ test_*.py (custom tier; gets the live_app fixture = the per-run domain). Remember shell-less images (e.g. static-web-server) can't be docker exec-ed — seed via the volume mountpoint like install_steps.sh does.

4. Verify cold on the cc-ci host

Commit + push the branch, then run the FULL unit suite cold against a fresh checkout on cc-ci:

git -C /tmp/cc-ci-feature add -A && git -C /tmp/cc-ci-feature commit -m "…" && git -C /tmp/cc-ci-feature push -u origin feat/<slug>
ssh cc-ci 'rm -rf /tmp/v && git clone -q "https://…@…/cc-ci.git" /tmp/v && cd /tmp/v && git checkout -q feat/<slug> && cc-ci-run -m pytest tests/unit/ -q'

Iterate until all unit tests pass. For UI changes, render the card/badge on cc-ci with cc-ci-run (use card.render_card_html / render_card_png / level_badge_svg) and eyeball the PNG.

5. Live end-to-end check (for anything touching run behaviour)

Run the harness on a representative recipe with FULL stages and inspect results.json:

ssh cc-ci 'cd /tmp/v && set -a; . /srv/cc-ci/.testenv; set +a; \
  CCCI_RUNS_DIR=/tmp/r RECIPE=<recipe> STAGES=install,upgrade,backup,restore,custom cc-ci-run runner/run_recipe_ci.py'

Confirm the tiers ran, the level/skips/badge are what you expect, teardown is clean (deploy-count == 1, no orphan stack/volume), and no secret leaked. Use a temp CCCI_RUNS_DIR so you don't pollute the dashboard. (custom-html-tiny is the lightest recipe for a smoke test.)

6. Open a PR (never merge directly)

Push the branch and open a PR via the Gitea API (POST /repos/recipe-maintainers/cc-ci/pulls, basic auth with GITEA_USERNAME:GITEA_PASSWORD). Tag @notplants for human awareness. Never push main.

7. Independent adversary verification

Write a verification plan (see cc-ci-plan/adversary-verify-pr6.md for the template) and dispatch an independent verifier (Agent tool, adversarial stance — disbelieve and verify; it must NOT reuse your working dirs). It does a fresh clone, the full unit suite cold, a diff regression review (especially: confirm no verdict/SSO enforcement was weakened, no test was weakened), a live full-stage run, non-vacuity checks, and teardown hygiene — returning VERDICT: PASS|REJECT with evidence.

8. Merge on PASS only

Only on an explicit adversary PASS, merge via the Gitea API (POST /repos/recipe-maintainers/cc-ci/pulls/<n>/merge, {"Do":"squash", …}) — squash keeps main clean. If REJECT, fix and re-verify.

9. Deploy to the running CI server

The merge lands on the cc-ci repo main, but the running harness is the deploy clone /root/builder-clone (NOTE: /root/cc-ci is gone — known deploy-path gap). Pull + rebuild:

ssh cc-ci 'cd /root/builder-clone && git pull --ff-only origin main && git submodule update --init --recursive && \
  nixos-rebuild switch --flake "/root/builder-clone?submodules=1#cc-ci"'

Verify the new code is present (grep for it in /root/builder-clone/runner/harness/…).

10. Clean up

Remove temp clones / run dirs on BOTH hosts (/tmp/cc-ci-feature locally, /tmp/v, /tmp/r on cc-ci).

Guardrails

• Never push main; never merge without an adversary PASS. The operator (or the adversary gate) owns merges.
• Cosmetics never change the verdict (R7). Level/card/badge/results.json are best-effort; the run's pass/fail (overall) is sovereign. Don't let a presentation change touch verdict logic.
• Presentation never inflates — an N/A (even an intentional, declared skip) caps the level; it is only labelled intentional, never promoted to a pass.
• Self-match pitfall: never pkill -f/pgrep -f a pattern that also matches your own ssh command line (it kills your session / loops forever). Target explicit PIDs instead.
• Push every cc-ci-orchestrator commit to git.autonomic.zone immediately (standing rule).