cc-ci-orchestrator/.claude/skills/ci-dev-workflow/SKILL.md

---
name: ci-dev-workflow
description: End-to-end workflow for developing a feature or fix in the cc-ci CI server (the harness, the level/results/card system, or a recipe's tests) from the orchestrator. Covers exploring the harness, branching a local clone, implementing with unit + live verification, opening a PR, getting an independent adversary verdict, merging, and deploying to the running CI server. Use whenever you are changing cc-ci server code (runner/harness/**, tests/**) rather than just running recipe upgrades. Invoke as /ci-dev-workflow.
---

# ci-dev-workflow

How to safely build and ship a change to the **cc-ci CI server** from the orchestrator. This is the
flow used to land the "intentional skips + 4-rung level ladder" feature. It never pushes `main` and
never merges without an independent adversary PASS.

## The two repos / two hosts
- **Orchestrator** (where you are): `/srv/cc-ci-orch` and `/srv/cc-ci` are both checkouts of the
  **cc-ci-orchestrator** repo (plans, skills, launchers). Edit `/srv/cc-ci-orch`; push to
  `git.autonomic.zone/recipe-maintainers/cc-ci-orchestrator`.
- **cc-ci CI server**: reached via `ssh cc-ci`. The CI server's own source is the **cc-ci** repo
  (`https://autonomic-bot:<token>@git.autonomic.zone/recipe-maintainers/cc-ci.git` — the token is in
  the `/root/builder-clone` remote and in `/srv/cc-ci/.testenv` as `GITEA_*`). The harness lives in
  `runner/harness/**`, the orchestrator entry is `runner/run_recipe_ci.py`, tests in `tests/**`.

> **Key constraint:** there is NO `python3`/`pytest` on the orchestrator host. Run all Python — unit
> tests, card/badge rendering, live harness runs — ON the cc-ci host via `ssh cc-ci`, using the
> `cc-ci-run` wrapper (it provides python + pytest + playwright/chromium + the harness env).

## Harness orientation (read before editing)
- `runner/harness/level.py` — PURE level ladder. **Cardinal invariant: presentation must never look
  greener than the tests.** The ladder is the four ESSENTIAL rungs `install · upgrade · backup_restore
  · functional` (top = L4); a gap (FAIL *or* N/A) caps the climb. integration/recipe-local are
  optional and NOT leveled. Keep this module pure + unit-tested.
- `runner/harness/results.py` — builds `results.json` (`derive_rungs`, `skips`, `build_results`).
  `skips` splits N/A rungs into intentional (declared in `recipe_meta.EXPECTED_NA={rung:reason}`) vs
  unintentional. Results assembly is best-effort (R7): a failure here NEVER changes the run verdict.
- `runner/harness/card.py` — summary card (HTML→PNG via headless chromium) + the level badge SVG.
- `runner/run_recipe_ci.py` — the orchestrator: deploys once, runs the tiers
  (`install,upgrade,backup,restore,custom`), computes the verdict. **The run VERDICT (SSO/F2-11,
  teardown, leak scan) is separate from the cosmetic level/card — never weaken verdict enforcement.**
- A recipe's tests live in `tests/<recipe>/`: `recipe_meta.py` (timeouts, `EXPECTED_NA`,
  `BACKUP_CAPABLE`, `DEPS`, …), `functional/` + `playwright/` (custom tier), `install_steps.sh`
  (pre-deploy hook), `ops.py` (pre-op seed hooks). The generic tiers are `tests/_generic/`.

## Steps

### 1. Explore first
Read the modules you'll touch + their unit tests (`tests/unit/test_*.py`) and any recipe under
`tests/<recipe>/`. Understand the verdict-vs-cosmetic split and the cardinal "never inflate" rule.
For a recipe-behaviour question, also look at the recipe on the host:
`ssh cc-ci 'cat ~/.abra/recipes/<recipe>/compose*.yml'`.

### 2. Branch a local clone of cc-ci (on the orchestrator)
So you can edit with real tooling (Read/Edit/Write) instead of ssh+heredocs:
```
cd /tmp && git clone "https://autonomic-bot:<token>@git.autonomic.zone/recipe-maintainers/cc-ci.git" cc-ci-feature
cd cc-ci-feature && git checkout -b feat/<slug>
git config user.name autonomic-bot && git config user.email autonomic-bot@git.autonomic.zone
```

### 3. Implement
Edit the harness + tests in the local clone. Match surrounding style. For pure logic (level/results),
add focused unit tests in `tests/unit/`. For recipe behaviour, add a `tests/<recipe>/functional/
test_*.py` (custom tier; gets the `live_app` fixture = the per-run domain). Remember shell-less
images (e.g. `static-web-server`) can't be `docker exec`-ed — seed via the volume mountpoint like
`install_steps.sh` does.

### 4. Verify cold on the cc-ci host
Commit + push the branch, then run the FULL unit suite cold against a fresh checkout on cc-ci:
```
git -C /tmp/cc-ci-feature add -A && git -C /tmp/cc-ci-feature commit -m "…" && git -C /tmp/cc-ci-feature push -u origin feat/<slug>
ssh cc-ci 'rm -rf /tmp/v && git clone -q "https://…@…/cc-ci.git" /tmp/v && cd /tmp/v && git checkout -q feat/<slug> && cc-ci-run -m pytest tests/unit/ -q'
```
Iterate until **all** unit tests pass. For UI changes, render the card/badge on cc-ci with `cc-ci-run`
(use `card.render_card_html` / `render_card_png` / `level_badge_svg`) and eyeball the PNG.

### 5. Live end-to-end check (for anything touching run behaviour)
Run the harness on a representative recipe with FULL stages and inspect `results.json`:
```
ssh cc-ci 'cd /tmp/v && set -a; . /srv/cc-ci/.testenv; set +a; \
  CCCI_RUNS_DIR=/tmp/r RECIPE=<recipe> STAGES=install,upgrade,backup,restore,custom cc-ci-run runner/run_recipe_ci.py'
```
Confirm the tiers ran, the level/skips/badge are what you expect, **teardown is clean** (deploy-count
== 1, no orphan stack/volume), and no secret leaked. Use a temp `CCCI_RUNS_DIR` so you don't pollute
the dashboard. (`custom-html-tiny` is the lightest recipe for a smoke test.)

### 6. Open a PR (never merge directly)
Push the branch and open a PR via the Gitea API (`POST /repos/recipe-maintainers/cc-ci/pulls`, basic
auth with `GITEA_USERNAME:GITEA_PASSWORD`). Tag `@notplants` for human awareness. **Never push `main`.**

### 7. Independent adversary verification
Write a verification plan (see `cc-ci-plan/adversary-verify-pr6.md` for the template) and dispatch an
**independent** verifier (Agent tool, adversarial stance — disbelieve and verify; it must NOT reuse
your working dirs). It does a fresh clone, the full unit suite cold, a **diff regression review**
(especially: confirm no verdict/SSO enforcement was weakened, no test was weakened), a live full-stage
run, non-vacuity checks, and teardown hygiene — returning `VERDICT: PASS|REJECT` with evidence.

### 8. Merge on PASS only
Only on an explicit adversary PASS, merge via the Gitea API
(`POST /repos/recipe-maintainers/cc-ci/pulls/<n>/merge`, `{"Do":"squash", …}`) — squash keeps `main`
clean. If REJECT, fix and re-verify.

### 9. Deploy to the running CI server
The merge lands on the cc-ci repo `main`, but the **running harness** is the deploy clone
`/root/builder-clone` (NOTE: `/root/cc-ci` is gone — known deploy-path gap). Pull + rebuild:
```
ssh cc-ci 'cd /root/builder-clone && git pull --ff-only origin main && git submodule update --init --recursive && \
  nixos-rebuild switch --flake "/root/builder-clone?submodules=1#cc-ci"'
```
Verify the new code is present (`grep` for it in `/root/builder-clone/runner/harness/…`).

### 10. Clean up
Remove temp clones / run dirs on BOTH hosts (`/tmp/cc-ci-feature` locally, `/tmp/v`, `/tmp/r` on cc-ci).

## Guardrails
- **Never push `main`; never merge without an adversary PASS.** The operator (or the adversary gate)
  owns merges.
- **Cosmetics never change the verdict (R7).** Level/card/badge/results.json are best-effort; the
  run's pass/fail (`overall`) is sovereign. Don't let a presentation change touch verdict logic.
- **Presentation never inflates** — an N/A (even an intentional, declared skip) caps the level; it is
  only *labelled* intentional, never promoted to a pass.
- **Self-match pitfall:** never `pkill -f`/`pgrep -f` a pattern that also matches your own ssh command
  line (it kills your session / loops forever). Target explicit PIDs instead.
- **Push every cc-ci-orchestrator commit** to git.autonomic.zone immediately (standing rule).