cc-ci-orchestrator/cc-ci-plan/plan-phase1e-harness-corrections.md

# cc-ci Phase 1e — Generic-harness corrections (Autonomous Build Plan)

**Status:** QUEUED — runs **after Phase 1d** and **before Phase 2** (`plan-phase2-recipe-tests.md`).
It corrects the **shared generic-test harness** from 1d, so it must land before Phase 2 authors
overlays on top of it.
**Transition:** **manual** (operator kicks it off).
**Builds on:** the Phase-1d generic suite (`runner/run_recipe_ci.py`, `runner/harness/*`,
`tests/_generic/*`, `tests/conftest.py`) — see `plan-phase1d-generic-test-suite.md`.
**Owner agents:** same Builder + Adversary loops (`plan.md` §6/§7); Adversary cold-verifies.
**This file's path:** `/srv/cc-ci/cc-ci-plan/plan-phase1e-harness-corrections.md`
**Phase order:** 1c → 1b → 1d → **1e** → 2 → 2b → 3.

---

## 0. Why this phase

An operator review of the 1d generic suite (2026-05-28) found three corrections to the **shared
harness** — the foundation every recipe overlay (Phase 2) builds on. Fixing them now, once, is far
cheaper than after overlays exist. All three are small in code but change behavior, so each needs a
fresh Adversary cold-verification and must not weaken any existing test.

---

## 1. Definition of Done (Phase 1e exit condition)

Terminates when every item holds **and the Adversary has independently cold-verified** (logged in
`machine-docs/REVIEW-1e.md`):

- [ ] **HC1 — Upgrade tier upgrades to the code under test (PR head), not a published tag.** The
      upgrade tier deploys the **previous published version** (last release before the PR) and then
      **upgrades to the PR head via `abra app deploy --chaos`** (chaos = the current checkout). The
      PR's actual changes are exercised by the upgrade path. (§2.1)
- [ ] **HC2 — Repo-local (PR-authored) code is not executed unless the recipe is approved.** By
      default the harness runs **only cc-ci-authored** overlays/install-steps (`tests/<recipe>/…`) +
      the generic; PR-authored repo-local `test_*.py` and `install_steps.sh` are **not run**.
      Repo-local code is honored **only for recipes on an explicit cc-ci-maintained approval
      allowlist** (default-deny). (§2.2)
- [ ] **HC3 — Generic runs by default (additive); skipping it is explicit.** When a recipe ships an
      overlay for an op, the **generic still runs** alongside it by default; the generic is skipped
      **only** when an explicit env/flag opts out. The baseline floor is never lost silently. (§2.3)
- [ ] **HC4 — No regression, cold-verified.** The Adversary re-runs the relevant D1–D10 / DG1–DG8
      acceptance from a cold start: nothing weakened, deploy-once (DG4.1) still holds, teardown still
      sacred, and the three new behaviors are demonstrated (HC1: a PR-head upgrade proven to deploy
      PR-head; HC2: a repo-local test is *ignored* for a non-approved recipe and *run* for an approved
      one; HC3: generic runs with an overlay present, and is skipped only with the opt-out set).

When HC1–HC4 hold and are confirmed, write `## DONE` to `machine-docs/STATUS-1e.md`.

---

## 2. The three corrections

### 2.1 HC1 — Upgrade to the PR head (not a published tag)
Current 1d behavior: deploy previous published version, then `abra app upgrade` to the **newest
published tag** — and because deploying the prev tag re-checks-out the recipe, the **PR-head code is
never deployed**, so a recipe PR's changes aren't exercised by upgrade.

Corrected:
1. Deploy the **previous published version** (the last release before the code under test) as the
   "before" state.
2. **Restore the PR-head checkout** (re-checkout the PR ref / re-use the post-fetch snapshot — the
   prev-tag deploy will have reset `~/.abra/recipes/<recipe>`).
3. **Upgrade to it via `abra app deploy --chaos`** (chaos = current checkout = PR head) in place on
   the shared deployment.
4. Assert reconverge + still serving (as today).
- **Adapt the "deployment moved" assertion** (`generic.do_upgrade`): prev→PR-head may *not* bump the
  coop-cloud version label (a PR can change a recipe without a version bump), so also accept an
  image/config change, or assert the running config now matches the PR head — keep it non-vacuous
  without false-failing a legit unbumped PR.
- **Non-PR `!testme`** (no PR head): "current checkout" = the catalogue current, so upgrade tests
  prev→current — still valid.
- Preserve **deploy-once** spirit: this is still one app deployment mutated in place (prev → chaos
  redeploy of PR head is the upgrade op, not a fresh second app). Reconcile with the DG4.1
  deploy-count guard — define whether a chaos redeploy counts as a "deploy" and adjust the guard so
  the legitimate upgrade isn't flagged (e.g. count `abra app new` installs, not in-place redeploys).

### 2.2 HC2 — Repo-local trust gate (default-deny; cc-ci overlays only)
`install_steps.sh` and repo-local `test_*.py` are PR-author-controlled code that runs on the CI host
with `/run/secrets/*` present — an untrusted-code risk. Operator decision (2026-05-28):

- **Default:** the harness runs **only cc-ci-authored** overlays + install-steps
  (`tests/<recipe>/…`) and the generic. Repo-local (`<recipe-repo>/tests/`) `test_*.py` and
  `install_steps.sh` are **discovered-but-not-executed**.
- **Approved recipes only:** repo-local code is honored **only** when the recipe is on an explicit,
  **cc-ci-maintained approval allowlist** (default-empty ⇒ default-deny). Adding a recipe to the
  allowlist is a deliberate cc-ci-maintainer act after reviewing that recipe's tests.
- Update `discovery.resolve_op` / `custom_tests` / `install_steps` so the **repo-local source is
  only consulted for allowlisted recipes**; otherwise precedence is **cc-ci > generic** only.
- **Open (settle in DECISIONS):** the allowlist's form + location (a checked-in file like
  `tests/repo-local-approved.txt`, or a field in a cc-ci config), and the approval workflow. Keep it
  simple + auditable + in git.
- (Future hardening, → IDEAS, not this phase: sandbox/network-restrict even cc-ci overlays.)

### 2.3 HC3 — Generic by default (additive), explicit opt-out
Supersedes 1d's pure-override default. New rule: when a recipe ships an overlay for an op, **both the
generic and the overlay run** for that op by default; the generic is skipped **only** when an
explicit opt-out is set.

- **Opt-out mechanism (propose; settle in DECISIONS):** an env flag `CCCI_SKIP_GENERIC` (all ops) and
  per-op `CCCI_SKIP_GENERIC_<OP>` (e.g. `..._UPGRADE`), settable via the recipe's `recipe_meta.py`
  (a `SKIP_GENERIC` list) so it's declarative per recipe, not a hidden global.
- **Op-vs-assertion split (required by additive + deploy-once):** a mutating op (upgrade/backup/
  restore) must run **once**, then **both** the generic assertions and the overlay assertions
  evaluate the post-op state — never upgrade/backup twice. So refactor the tiers: the **orchestrator
  performs the op once** (the harness owns the op), then runs generic assertions (unless opted out) +
  overlay assertions against the shared post-op deployment. For `install` (no op) both assertion sets
  just run. This keeps deploy-once and one-op-per-tier intact.
- Net effect: the generic "is it actually serving / did the upgrade move / snapshot produced" floor
  is **always** exercised unless a recipe explicitly declares it skips generics — overlays add, they
  don't silently subtract.

---

## 3. Method / milestones (bounded)
- **E0 — HC2 trust gate.** Gate repo-local behind the approval allowlist (default-deny); cc-ci+generic
  only otherwise. *Accept:* repo-local ignored for a non-approved recipe, run for an approved one.
- **E1 — HC3 additive + op/assertion split.** Generic runs alongside overlays by default; op runs
  once; opt-out env skips the generic assertions. *Accept:* overlay + generic both run on one
  deployment; opt-out skips generic; deploy-count still 1.
- **E2 — HC1 upgrade-to-PR-head.** prev-release → PR-head via `deploy --chaos`; moved-assertion
  adapted; deploy-count guard reconciled. *Accept:* upgrade demonstrably deploys PR-head.
- **E3 — HC4 cold re-verification + docs.** Adversary cold-verifies no regression + the three new
  behaviors; update `docs/` + `machine-docs/DECISIONS.md`; flip `STATUS-1e.md` to `## DONE`.

---

## 4. Guardrails
- **Never weaken a test** — these are correctness/security fixes; the cardinal rule still wins.
- **Default-secure** — repo-local PR code is off unless the recipe is explicitly approved; the
  allowlist lives in git and is auditable.
- **Floor-by-default** — the generic baseline always runs unless a recipe explicitly opts out.
- **Deploy-once preserved** — one app deployment, one teardown; ops run once; reconcile the DG4.1
  guard with the chaos-upgrade redeploy.
- **Bounded** — three fixes + verification, then stop; bigger hardening (sandboxing) → IDEAS.

## 5. Open decisions (log in machine-docs/DECISIONS.md)
- HC2: approval-allowlist form/location + the approval workflow.
- HC3: opt-out flag name/granularity + declaring it via `recipe_meta.py`.
- HC1: how the DG4.1 deploy-count guard treats an in-place chaos upgrade (don't flag the legit op).