cc-ci-orchestrator/cc-ci-plan/plan-phase2w-warm-canonical-quick.md

# cc-ci Phase 2w — Warm canonical deployments + `--quick` CI mode (Autonomous Build Plan)

**Status:** ACTIVE — **interjected into Phase 2** by operator decision (2026-05-28). Phase 2
(`plan-phase2-recipe-tests.md`) is **PAUSED at its current progress** (its STATUS-2/BACKLOG-2 state is
preserved); the loops do this phase now, then **Phase 2 resumes automatically** where it left off.
**Transition:** auto — on `## DONE` in `machine-docs/STATUS-2w.md` the watchdog returns to Phase 2.
**Builds on:** the Phase-1d/1e harness (generic suite, deploy-once, override overlays, HC1 upgrade
to PR-head, the sso-dep pattern `plan-sso-dep-testing.md`) and the now-wired Docker Hub auth.
**Owner agents:** Builder + Adversary loops (`plan.md` §6/§7); Adversary cold-verifies.
**This file:** `/srv/cc-ci/cc-ci-plan/plan-phase2w-warm-canonical-quick.md`
**Phase order now:** 1c → 1b → 1d → 1e → 2(paused) → **2w** → 2(resume) → 2b → 3 → 4.

---

## 0. Why this phase

Cold-start CI (fresh `abra app new` → deploy → DB-init/first-boot → … → teardown) is slow, and it
re-pays that cost on every run and for every SSO dependency. This phase adds a **warm-data** layer:
keep each app's **data volume** around between runs (Co-op Cloud's `undeploy` frees RAM but keeps
volumes), so a fast `--quick` run can reattach it, upgrade to the PR code, and assert — without the
cold-provisioning cost. A persistent **keycloak** serves SSO-dependent recipes without a fresh
co-deploy each run. A **last-known-good snapshot per app** means a bad PR tested under `--quick` can
never destroy the working state+data — we roll back.

**Terminology (use these terms throughout code/docs/decisions):**
- **live-warm** — actually deployed and running (e.g. keycloak): instant to use, costs RAM.
- **data-warm** — *undeployed* (RAM freed) but its **data volume is retained**, so a later
  `abra app deploy` reattaches it and boots warm (skips fresh DB-init/first-boot), costs only disk.
- **cold** — no retained data: fresh `abra app new` + new volume + full lifecycle + teardown that
  deletes the volume. The authoritative default.

**Design principles settled with the operator (do not relitigate):**
- **Keep keycloak live-warm; keep everything else data-warm.** Only keycloak (shared dep) + the one
  app under test run at a time. RAM stops being the limiter; **disk is the budget** (monitor; bump
  only if needed — test fixtures are small).
- **Default `!testme` = full cold** (authoritative; its upgrade tier already exercises PR-upgrade per
  1e). **`--quick` is an opt-in flag**, a lower-confidence fast lane.
- **The canonical known-good advances ONLY via cold runs** (esp. the nightly sweep). `--quick` NEVER
  promotes the canonical — it consumes it read-mostly and rolls back on failure.
- **Snapshots: raw volume copy taken while UNDEPLOYED** (fast + consistent because nothing is
  writing). **One last-known-good per app.**
- Warm volumes + snapshots are **cache, not source** — not in the git/D8 closure; re-seeded by cold
  runs, not restored on a VM rebuild.
- **Warm/infra apps (traefik + keycloak) auto-update to LATEST, nightly, with health-gated
  rollback** (operator, 2026-05-28). Both are **unpinned** — their reconcilers `abra recipe fetch`
  the latest published recipe + chaos-deploy it. A **nightly `nixos-rebuild switch`** runs the
  reconcilers so the warm/infra apps roll to latest each night. Because the recipe is fetched at
  *activation* (runtime), the **nix closure stays byte-identical** (only the deployed versions float)
  — D8 is preserved; the version pin is gone, so the closure is *more* stable, not less.
- **Auto-update is gated TWICE — a pre-deploy safety gate AND a post-deploy health gate:**
  - **Pre-deploy (don't even try unsafe upgrades):** only **auto-apply upgrades that don't require
    manual intervention** — i.e. non-major (patch/minor) recipe-version bumps with no
    manual-migration in their release notes. If current→latest is a **MAJOR bump** or the target's
    **release notes flag a manual migration**, **DO NOT auto-upgrade**: stay on the current version
    and **PUSH ALERT** with the release notes (operator upgrades manually). A passing health check
    does NOT prove a required migration was done, so this gate is independent of health. Lean on
    abra's own major-bump caution + the recipe `releaseNotes/`.
  - **Post-deploy (for upgrades we DO apply):** record running version as last-good → deploy latest
    → health-check → **healthy: commit (last-good := latest); unhealthy: roll back to last-good +
    PUSH ALERT.** For **stateful apps (keycloak / any app with a DB/volume): snapshot the data volume
    BEFORE the upgrade and restore it on rollback** (a forward DB migration can make a version-only
    rollback fail) — reusing the WC3 known-good-snapshot mechanism. traefik (stateless) = version
    rollback only. (Rollback is NOT nix-generation rollback — the swarm app isn't in the generation;
    it's built into the reconciler.)

---

## 1. Definition of Done (Phase 2w exit condition)

Terminates when every item holds **and the Adversary has independently cold-verified** (logged in
`machine-docs/REVIEW-2w.md`):

- [ ] **WC1 — Live-warm keycloak (SSO dep), unpinned + self-healing.** A persistent (live-warm)
      keycloak runs at a stable domain, **unpinned** (reconciler `abra recipe fetch` latest +
      chaos-deploy, matching traefik — drop the `kcVersion` pin; keep the *secret-generate-only-if-
      missing* guard + the health-wait). SSO-dependent recipes (per `plan-sso-dep-testing.md`) point
      their `setup_custom_tests` at it, create a **per-run namespaced realm+client**, then **delete
      that realm** after the run, instead of co-deploying a fresh keycloak. Proven: a dependent
      recipe's SSO custom tests pass against the warm keycloak; concurrent dependents don't collide
      (distinct realms); leftover realms are reaped.
- [ ] **WC1.2 — Pre-deploy auto-upgrade safety gate (manual-migration → alert, don't auto-apply).**
      Before the reconciler auto-applies "latest", it checks the current→latest delta: **auto-apply
      only non-major (patch/minor) bumps with no manual-migration release notes.** A **MAJOR
      recipe-version bump**, or a target whose **`releaseNotes/` flag a manual migration**, is NOT
      auto-applied — the reconciler **stays on the current version and pushes an alert with the
      release notes** for the operator to upgrade manually. (Health-pass ≠ migration-done, so this is
      independent of WC1.1.) Detection leans on abra's major-bump handling + the recipe release notes.
      **Adversary proof:** simulate a major/manual-migration latest → confirm the warm app stays on
      current + an alert with the notes fired (no silent auto-upgrade).
- [ ] **WC1.1 — Health-gated deploy-with-rollback in the warm/infra reconcilers (traefik + keycloak).**
      Each reconciler: record the running version as **last-good** → fetch+deploy latest →
      health-check → **healthy: commit last-good := latest; unhealthy: roll back to last-good +
      `PushNotification` alert.** For **stateful apps (keycloak): snapshot the data volume BEFORE the
      upgrade; on rollback restore that snapshot + redeploy the prior version** (forward DB migrations
      make version-only rollback unsafe) — reuse the WC3 snapshot helper. traefik (stateless) = version
      rollback only. **Adversary proof:** force a broken "latest" (simulate) → confirm the warm app
      self-reverts to the prior healthy version (keycloak with its pre-upgrade data intact) and an
      alert fired; a healthy update commits the new version as last-good.
- [ ] **WC2 — Data-warm canonical model.** A canonical per warmed recipe at a **stable domain**
      (distinct from cold per-run `<recipe>-<6hex>` domains), kept **data-warm** (undeployed-when-idle,
      volume retained). A small declarative registry/reconciler tracks which recipes are
      canonical and **at which commit** their known-good is. Re-warmable from scratch (cache).
- [ ] **WC3 — Known-good snapshots.** For each canonical app, a **raw copy of its data volume(s)
      taken while undeployed**, stored under a stable path (e.g. `/var/lib/ci-warm/<recipe>/`),
      tagged with the commit it passed on. **One last-known-good retained per app** (prior is
      replaced atomically on update). Restore is proven to bring the app back healthy with its data.
- [ ] **WC4 — `--quick` mode.** `runner/run_recipe_ci.py` gains a `--quick` path (flag/env): reattach
      the canonical warm volume (`abra app deploy` of the canonical) → **upgrade to PR head** (chaos
      redeploy) → run generic UPGRADE + serving + custom assertions (generic-first invariant holds) →
      **on PASS:** `abra app undeploy` (keep volume), do NOT alter the known-good; **on FAIL:**
      restore the last-known-good snapshot, then undeploy. `--quick` **never promotes** the canonical.
- [ ] **WC5 — Canonical advancement via cold only.** A **green full-cold run on latest** re-snapshots
      + re-tags the canonical known-good (promote-on-green instead of deleting at teardown). A cold
      run is the ONLY thing that advances a canonical. Seeding: the first green cold run on latest
      makes an app canonical.
- [ ] **WC6 — Nightly: rebuild (warm/infra → latest) THEN full-cold sweep.** A scheduled nightly job,
      in order: (1) **`nixos-rebuild switch`** → the warm/infra reconcilers roll traefik + keycloak to
      latest, subject to the **WC1.2 pre-deploy gate** (major/manual-migration → hold on current +
      alert) and the **WC1.1 health-gated rollback**; (2) the **full cold** suite across enrolled recipes
      — refreshing every canonical's known-good (WC5) AND serving as a daily authoritative regression
      run, now against the freshly-updated infra. Don't run while a test run is in flight. Mechanism
      settled in DECISIONS (systemd timer on cc-ci / Drone cron / bridge), declarative + reproducible.
      Bounded by MAX_TESTS (serial is fine — nightly). If the rebuild's health-gate rolled an infra
      app back, the alert fires and the sweep still runs against the (healthy) prior version.
- [ ] **WC7 — Trigger + authority + labeling.** Default `!testme` = full cold (unchanged). `--quick`
      is opt-in (`!testme --quick`, or a build param) and **never gates merge**. Run results carry
      the **mode** (cold vs quick) so a `--quick` pass is distinctly labeled lower-confidence (feeds
      Phase 3). Quick requires an existing canonical; if none, it cleanly falls back to (or reports
      "no canonical — run cold first").
- [ ] **WC8 — Resource safety + isolation.** Warm-base runs serialize per app (MAX_TESTS honored);
      warm keycloak shared safely via per-run realms; **disk monitored** (warm volumes + one snapshot
      each) with a documented budget + prune of stale/orphaned warm data; cold-run teardown stays
      sacred (deletes its own per-run volumes); warm data is excluded from the D8 reproducibility
      closure (documented as cache).
- [ ] **WC9 — Documented + cold-verified, incl. the rollback proof.** `docs/` explains warm/quick;
      the Adversary cold-verifies, **including deliberately failing a PR under `--quick` and
      confirming the canonical's last-known-good is restored intact (data preserved)**, and that a
      `--quick` pass did not move the known-good. No softened tests.

When WC1–WC9 hold and are confirmed, write `## DONE` to `machine-docs/STATUS-2w.md` → the watchdog
auto-returns to **Phase 2** (resume recipe authoring).

---

## 2. The `--quick` flow (reference)

```
PRECOND: a canonical for <recipe> exists (seeded by a prior green cold run); else fall back/repor​t.
  1. abra app deploy <canonical-domain>      # reattach warm volume -> fast warm boot at known-good commit
  2. wait_healthy
  3. (deps) point at the warm keycloak; create a per-run realm+client (namespaced)
  4. UPGRADE to PR head (abra app deploy --chaos to the PR checkout)   # the op, once
  5. assert: generic upgrade (reconverge + moved + serving) + recipe overlay + custom (requires_deps)
  6a. PASS -> abra app undeploy <canonical-domain>      # keep volume; known-good UNCHANGED
  6b. FAIL -> restore last-known-good snapshot to the volume; abra app undeploy   # roll back, data safe
  7. (deps) delete the per-run realm from the warm keycloak
```
Cold run (default, unchanged) seeds/advances the canonical: on a green cold run on latest, snapshot
the (undeployed) volume → replace the last-known-good + tag the commit, and keep the volume as the
new canonical instead of deleting it.

## 3. Milestones (bounded)
- **W0 — Warm keycloak, unpinned + self-healing (WC1, WC1.1).** Highest ROI; unblocks faster SSO
  recipe tests for the resumed Phase 2. Includes the health-gated deploy-with-rollback (snapshot
  keycloak before upgrade, restore on health-fail + alert); apply the same to traefik (version-only).
- **W1 — Canonical registry + snapshot/restore (WC2, WC3).** Stable-domain warm apps; raw-while-
  stopped snapshot + restore; prove restore round-trips data. (Shares the snapshot helper with WC1.1.)
- **W2 — `--quick` mode (WC4, WC7).** Orchestrator path + labeling + fallback.
- **W3 — Nightly rebuild→sweep + cold-advances-canonical (WC5, WC6).** Nightly `nixos-rebuild`
  (warm/infra → latest, health-gated) then full-cold sweep; promote-on-green-cold; scheduled job.
- **W4 — Resource/isolation hardening + docs + cold verify incl. rollback proof (WC8, WC9).** Then
  `## DONE`.

## 4. Guardrails
- **`--quick` never advances the canonical; only cold does.** Anchors the baseline to verified states.
- **Never lose the known-good** — snapshot before mutate (or rely on the standing known-good); restore
  on any quick failure. The rollback proof (WC9) is mandatory.
- **Default stays cold; quick is opt-in + clearly lower-confidence.** Don't let a quick pass read as
  full confidence.
- **Snapshot only while undeployed** (consistency). **One last-known-good per app** (disk).
- **Cold teardown stays sacred** (deletes per-run volumes); warm volumes are a managed cache, never
  confused with per-run state; warm data excluded from D8.
- **Warm/infra auto-update is health-gated** — a failed "latest" self-reverts to the last-good
  version (+ data, for stateful apps) and alerts; never leave the proxy/SSO dep broken silently.
- **Never weaken a test** (cardinal rule). Generic-first invariant holds in `--quick` too.
- **Bounded** — build the mechanism + prove on keycloak + a couple of recipes; do NOT re-warm all
  recipes here (the nightly sweep populates canonicals over time).

## 5. Open decisions (log in machine-docs/DECISIONS.md)
- Canonical **stable-domain scheme** (distinct from cold per-run domains) + how the registry/reconciler
  is declared.
- **Snapshot storage + format** (raw tar vs reflink/CoW copy) under `/var/lib/ci-warm/`; atomic replace.
- **Nightly sweep mechanism** (systemd timer / Drone cron / bridge) + ordering + disk-prune policy.
- `--quick` **trigger surface** (`!testme --quick` comment vs Drone build param) + the "no canonical
  yet" fallback (run cold vs report-and-skip).
- **Disk budget**: measure warm volume + snapshot sizes across recipes; decide if a 30→larger bump is
  needed or the warm set stays bounded.
- **Stateful pre-upgrade snapshot consistency (keycloak).** keycloak is *live-warm* (running) at
  nightly-upgrade time, but the snapshot rule is "raw copy while UNDEPLOYED." Cleanest: the nightly
  keycloak update = **undeploy → raw snapshot → deploy latest → health-check → on fail restore
  snapshot + redeploy prior** (the brief nightly downtime makes the snapshot consistent and honors
  the WC3 invariant). Confirm this vs an app-consistent backup alternative.
- **last-good version state** for warm/infra apps (where the reconciler records the prior healthy
  version to roll back to) — a small state file alongside the snapshot, re-derivable from the running
  swarm version label.
- **Manual-migration / major-bump detection (WC1.2).** How to decide "auto-apply vs alert-and-hold":
  primary signal = **major recipe-version bump** (coop-cloud `<upstream>+<recipe-semver>`; major
  recipe-semver = breaking, matches abra's own major-upgrade caution); secondary = scan the target's
  `releaseNotes/<version>.md` for manual-migration markers. Decide the exact rule + whether to parse
  `abra app upgrade` output vs compute the delta directly.