cc-ci/machine-docs/JOURNAL-2w.md

# JOURNAL — Phase 2w (warm canonical + `--quick`) — Builder

Append-only reasoning log (WHY). Facts/verification go in STATUS-2w; verdicts in REVIEW-2w.

## 2026-05-28 — Phase 2w bootstrap + cleanup + W0 design

**Orientation.** Operator interjected Phase 2w into Phase 2 (Phase 2 paused, state preserved).
Read the 2w plan + plan.md §6.1/§7/§9. Adversary already online (REVIEW-2w `@2026-05-28 start`),
idle awaiting a WC gate claim. Seeded STATUS-2w/BACKLOG-2w/JOURNAL-2w.

**In-flight Phase 2 work committed.** Working tree had an uncommitted edit to
`tests/lasuite-drive/setup_custom_tests.sh` (Q3.2 MinIO bucket creation via the createbuckets
one-shot) — the continuation of commit 6557197. Committed it (66e065d) with an honest message: not
yet live-verified (needs a lasuite-drive deploy once warm keycloak exists). This preserves Phase 2
progress at the pause point; it resumes after 2w DONE.

**Cleanup (orchestrator-requested).** cc-ci `/` was at 91% (only 2.4G free) — a real WC8 concern
before adding warm volumes/snapshots. Tore down the leftover COLD per-run apps from paused Phase 2
via `lifecycle.teardown_app(..., verify=True)`: `lasu-0a6fb2` (12-service lasuite-drive, heaviest),
`keyc-07d81e` (cold keycloak), `lasu-dbg` (debug lasuite). All TEARDOWN OK, no residual. Disk →
86% (3.8G free). Only infra stacks remain (backups, bridge, dashboard, drone, traefik). Did NOT
`docker image prune` — 9.7GB reclaimable but the image cache is the warm pull-cache; with authed
Docker Hub pulls now wired, a re-pull is billed to the account (cheaper) but still slow, so keep the
cache. Disk is the Phase-2w budget (WC8) — monitor.

**W0 design (WC1 — live-warm keycloak).** The existing SSO harness is already most of the way there:
- `sso.setup_keycloak_realm(provider_domain, realm, client_id, ...)` creates a realm+client+user
  **idempotently via the admin API**, and `_kc_admin_password` reads the admin password from inside
  the running container (`docker exec ... cat /run/secrets/admin_password`). So it works against ANY
  running keycloak — cold or warm — with no external password handling.
- The orchestrator dep flow (`run_recipe_ci.py`): `declared_deps` → `deploy_deps` (fresh co-deploy
  per run) → `_enrich_deps_with_sso` (creates realm, realm name currently = `parent_recipe`) →
  `setup_custom_tests.sh` hook → teardown_deps (undeploy).

What WC1 changes:
1. The **realm becomes the per-run isolation unit** on a shared live-warm keycloak. Realm name must
   be unique per (parent, pr, ref) so concurrent dependents don't collide — change from
   `realm=parent_recipe` to `realm=<parent>-<6hex>` (derive the hex from the parent's per-run domain
   label so it's stable within a run and distinct across concurrent runs).
2. The keycloak dep is **not co-deployed**: point at the stable warm domain; on teardown **delete the
   realm** (not undeploy keycloak). Fall back to cold co-deploy if no warm keycloak is present (so a
   from-scratch / no-warm environment still works — the warm keycloak is an optimization layer).
3. The warm keycloak itself is **declarative infra** (Nix reconciler, like traefik) — NOT warm
   *data* (so it IS in the D8 closure as a reconciler; its realm data is ephemeral per-run anyway).
   Re-warmable from scratch.

Stable-domain scheme decision: `warm-<recipe>.ci.commoninternet.net` (here `warm-keycloak...`),
clearly distinct from cold `<recipe[:4]>-<6hex>`. Risk: longer stack name → swarm 64-char
config/secret limit; will verify on first deploy and shorten if it overflows.

Building W0 in increments (each verified): (1) sso realm lifecycle prims + units; (2) deploy warm
keycloak manually at the stable domain and prove realm create→delete via admin API; (3) wire the
orchestrator live-warm mode; (4) declarative Nix reconciler; (5) e2e + concurrency + reaping proof.
</content>

## 2026-05-29 — W0 core mechanism PROVEN; declarative reconciler up; design update absorbed

**Stale Phase-2 run killed.** Found an orphaned `run_recipe_ci.py` (RECIPE=lasuite-drive, the Q3.2
`ccci-q32-drive-sso2.log` run) still alive from before the phase switch (PPID 1, nohup). It had
deployed lasu-0a6fb2 + tried a cold keyc-07d81e dep — both of which I'd already torn down, so it was
failing. Killed its process tree + janitored. Only infra + warm-keycloak remain.

**W0.1 realm lifecycle (sso.py)** — list_realms / delete_keycloak_realm (idempotent, refuses master)
/ realms_to_reap (pure predicate) / reap_orphaned_realms. +8 unit tests. The per-run realm is the
isolation unit on a shared keycloak; orphans reaped by hex-not-in-live-stacks (concurrency-safe).

**W0.2 orchestrator live-warm mode** — warm.py (stable-domain scheme, is_warm_up probe,
live_app_hexes, realm_for=<parent>-<6hex>, reap_orphan_realms). run_recipe_ci splits declared deps
into warm (shared provider + per-run realm, no deploy, realm deleted at teardown) vs cold
(co-deploy), warm only if provider up else cold fallback; deploy-count excludes warm deps; reaps
orphans at run start. Dependent tests now assert the namespaced realm pattern (stronger than ==parent).

**WC1 CORE MECHANISM PROVEN** (deploy-free, live warm keycloak): realm create → password-grant JWT
→ discovery issuer → delete(idempotent) → reap(keeps live hex, deletes orphan): ALL PASS.

**W0.3 declarative reconciler** (nix/modules/warm-keycloak.nix) — systemd oneshot, converges warm
keycloak. Two bugs found+fixed against the real system:
1. `abra app deploy` non-chaos FATALs "already deployed" → need `-f` (tested: redeploys at ENV
   VERSION, exit 0).
2. **Newline bite** (the backupbot.nix bite): keycloak's .env.sample ends with a newline-less
   `#COMPOSE_FILE=` comment, so bash `set_env`'s printf glued `DOMAIN=` onto that comment →
   DOMAIN unset → `KC_HOSTNAME=https://` (empty host) → keycloak crash-loop ("Expected authority at
   index 8: https://"). Fixed set_env to ensure a trailing newline before append (same as backupbot).
Also made converge **skip the redeploy when already 200** (no JVM-restart blip on every rebuild;
only (re)deploys when down/crash-looping). Verified: nixos-rebuild switch → warm-keycloak.service
active "no-op converge", system running (0 failed), /realms/master=200.

**W0.4 e2e (lasuite-docs vs warm keycloak)** — the WARM MECHANISM worked: deploy-count=1 (keycloak
NOT co-deployed), per-run realm `lasuite-docs-9c1995` created + **deleted on the warm keycloak** at
teardown, install pass. BUT `setup_custom_tests.sh exited 1` → 3 requires_deps SSO tests SKIPPED →
F2-11 correctly FAILED the run (not green). Root cause = a **lasuite-docs recipe race**, NOT warm
keycloak: the in-place `abra app deploy --force --chaos` (OIDC wiring) rolls all services; nginx
`web` fatally exits on `[emerg] host not found in upstream ...backend:8000` while backend is
mid-restart, and abra's converge check times out → "deploy failed 🛑". This is independent of
warm/cold keycloak (Q2.4 cold-keycloak lasuite-docs passed before; warm should REDUCE contention).
Filed as a finding to investigate (flaky/timing/resource vs deterministic regression); the headline
WC1 "dependent SSO tests green against warm keycloak" needs this resolved or a more-robust dependent.

**DESIGN UPDATE absorbed (orchestrator + Adversary REVIEW-2w, 2026-05-28→29).** Warm/infra apps
(traefik + keycloak) now AUTO-UPDATE to LATEST nightly with HEALTH-GATED ROLLBACK:
- **WC1 revised:** UNPIN keycloak (match traefik: `abra recipe fetch` latest + chaos deploy; DROP
  kcVersion). Keep secret-generate-only-if-missing + health-wait. D8 preserved (recipe fetched at
  runtime → nix closure byte-identical).
- **WC1.1 NEW:** health-gated deploy-with-rollback IN the reconcilers. record last-good → deploy
  latest → health-check → healthy: commit last-good:=latest; unhealthy: rollback + PushNotification.
  Stateful (keycloak): undeploy → raw snapshot data volume → deploy latest → on fail restore snapshot
  + redeploy prior version (forward DB migrations make version-only rollback unsafe). traefik
  (stateless) = version rollback only. Reuse WC3 snapshot helper.
- **WC1.2 NEW:** pre-deploy safety gate — auto-apply only non-major/no-manual-migration bumps; a
  MAJOR bump or manual-migration release notes → stay on current + alert (don't auto-apply).
- **WC6 reordered:** nightly = nixos-rebuild switch FIRST (warm/infra→latest, health-gated) THEN
  full-cold sweep; never while a test is in flight.

**Re-sequencing consequence:** WC1.1 depends on the **WC3 snapshot/restore helper**, so I build that
FIRST (foundational), then rewrite the reconciler ONCE into the full unpinned + health-gated +
safety-gated + rollback form (avoids reworking the reconciler twice). Current reconciler (pinned,
skip-if-healthy) is INTERIM — keeps keycloak live-warm/healthy meanwhile; will be replaced. Also need
to settle the **alert mechanism**: a bash systemd reconciler can't call the agent's PushNotification
tool directly — decision needed (alert sentinel file the Builder loop reads + relays, or a webhook).

## 2026-05-29 — W0.5 WC3 snapshot helper proven; disk reclaim (WC8 hygiene)

W0.5 warmsnap.py landed + LIVE round-trip proven on warm keycloak (see STATUS-2w). Then settled the
W0.6 reconciler approach (python entrypoint in nix store; deploy-by-tag; recipe-semver = pre-`+`
component) in DECISIONS.

**Disk reclaim.** After 3 nixos-rebuild switches + 3 keycloak deploy cycles (WC3 proof) + a 159M
keycloak snapshot, `/` hit 96% (1.2G free) — a WC8 red flag before continuing. Reclaimed safely
(reversibility is via the git-declared config, not old generations): `rm -rf /root/cc-ci.prev`;
`nix-collect-garbage -d` (2553 paths, 3.38G); `docker image prune -f` dangling-only (3.32G, KEEPS the
tagged pull-cache); pruned old abra deploy logs (keep last 5). Result: **62% (10G free)**. This
GC+dangling-prune is the disk-management mechanism WC8 must formalize (run it in the nightly/W4, and
keep one last-good snapshot per app bounded). NOTE for WC8: the WC3 keycloak snapshot is 159M; a
warm-set of ~6 canonicals × (volume + 1 snapshot) is the disk budget to size.

**State at checkpoint:** warm keycloak healthy (200), only infra+warm stacks, system running (0
failed), disk 62%. W0.1-W0.5 done+proven+pushed (HEAD 67240dc). Next unit: W0.6 reconciler rewrite
(unpin + WC1.2 safety gate + WC1.1 health-gated rollback), then W0.7/W0.8 (lasuite-docs race +
headline WC1 e2e).

## 2026-05-29 — W0.9 WC1.1 live proofs PASS (healthy upgrade + marquee rollback)

Built `runner/warm_reconcile.py`'s health-gated rollback and proved it live against the warm keycloak
using annotated fake tags + `CCCI_SKIP_FETCH=1`. The proof iterations surfaced 4 real issues, each
fixed against the real system (verify-don't-assume):

1. **deploy-failure must roll back too** — a broken "latest" can fail abra's *lint/converge*
   (deploy_version raises) rather than deploy-then-be-unhealthy; wrapped the upgrade deploy so BOTH
   raise and unhealthy paths trigger the snapshot-restore rollback (else the unit just crashes).
2. **warmsnap clobbered last_good** — snapshot's atomic swap renamed the whole `<recipe>/` dir,
   wiping the sibling `last_good` file. Fixed: snapshot lives in `<recipe>/snapshot/`; only that
   subdir is swapped; `last_good` (sibling) survives.
3. **swarm settle race** — abra undeploy returns before swarm finishes removing tasks, so an
   immediate snapshot/restore/redeploy of the same stack raced a half-removed stack. Added
   `wait_undeployed()` after every undeploy.
4. **abra writes FATA to stdout** — deploy_version only surfaced stderr (empty); now includes stdout.
   This is how I diagnosed the two test-artifact failures: the broken deploy failed abra **lint R009**
   (bad env not a string — a valid "broken latest"), and the first rollback attempts failed abra
   **lint R014 "only annotated tags used for recipe version"** because my fake tags were *lightweight*
   (production tags are annotated) — a TEST artifact, not a reconciler bug. Fixed the test to create
   annotated tags (peel `^{}` to avoid nested-tag; set git identity).

**Final PROOF (ALL PASS):**
- (a) healthy upgrade 10.7.1→10.7.9: snapshot taken (subdir), deploy, health-pass, last_good
  committed=10.7.9, marker realm preserved through the undeploy/snapshot/redeploy.
- (b) marquee rollback: broken latest 10.7.10 → deploy fails → rollback to 10.7.9 → HEALTHY; marker
  realm INTACT (data preserved through broken-upgrade + snapshot-restore); last_good NOT advanced;
  rollback alert sentinel written (attempted=10.7.10, last_good=10.7.9, recovered=True). keycloak
  recovered to canonical 10.7.1+26.6.2 healthy, no fake tags left.

This satisfies the WC1.1 Adversary mandate (broken latest → self-revert + data intact + alert;
healthy update commits last-good). WC1.2 holds were proven in W0.6. **The reconciler-side WC1/WC1.1/
WC1.2 are proven; the alert RELAY (Builder loop scans /var/lib/ci-warm/alerts/ → PushNotification +
archive to seen/) is still to wire (flagged for when nightly WC6 lands / a real alert can occur).**

Remaining for the WC1 gate: W0.7 (lasuite-docs in-place chaos-redeploy nginx race) + W0.8 (headline
dependent-SSO-green e2e vs warm keycloak + concurrent distinct realms + reaping).