# DECISIONS — cc-ci Builder

Architecture decisions and dead-ends. One line of rationale each. (§0, §8)

## Settled

- **nixos-rebuild submodule protocol — SETTLED (2026-06-13, phase pvfix).** The canonical nixos-rebuild command on the live host is `nixos-rebuild switch --flake "git+file:///root/builder-clone?submodules=1#cc-ci"`. The `path:` scheme does NOT support `?submodules=1` in this Nix version; `git+file://` does. Plain `nixos-rebuild switch --flake /root/builder-clone#cc-ci` fails with `secrets/secrets.yaml does not exist` because the git submodule is not included in the nix store copy.

- **deploy-proxy health gate — SETTLED (2026-06-13, phase pxgate, supersedes pvfix workaround).** Changed the traefik health probe from `ci.commoninternet.net/` (dashboard, ordered After=deploy-proxy → circular on cold boot) to `traefik.ci.commoninternet.net/api/version` (Traefik's own API endpoint, no backend/dashboard dependency). A broken traefik still fails the gate (returns non-200 or times out), so rollback semantics are preserved. Controlled reproduction confirms: with dashboard scaled to 0, old probe returns 404, new probe returns 200. Cold-boot deadlock eliminated. DEFERRED item 2026-06-13 closed by this fix. (Old pvfix note about concurrent manual restart workaround is now superseded.)

- **cfold deprecated-folder policy — SETTLED (2026-06-12, phase cfold).** `tests/<recipe>/custom/`
  is the canonical home for custom tests. Discovery keeps recognizing legacy `functional/` and
  `playwright/` subdirs for both cc-ci and approved repo-local tests as a temporary compatibility
  alias, but it emits a one-line warning to stderr whenever it discovers tests there. Rationale:
  the phase plan forbids silent coverage loss, and recipe repos outside this clone may still be on
  the old layout during the migration window.

- **Wildcard TLS:** operator pre-issues wildcard cert at `/var/lib/ci-certs/live/`; Traefik file
  provider serves it; **no ACME** for commoninternet.net. (Plan §4.0/§8 — fixed.)
- **Repo:** `git.autonomic.zone/recipe-maintainers/cc-ci`, private. Bot is org admin. (Bootstrap.)
- **Git credentials:** helper script in repo-local git config sources `/srv/cc-ci/.testenv` at call
  time — no secret values stored in `.git/config` or commits.

- **Proxy: real coop-cloud/traefik via abra — SETTLED (M1, orchestrator decision 2026-05-26,
  overrides plan §3 `modules/traefik.nix`).** Instead of a hand-rolled Traefik we deploy the
  canonical Co-op Cloud `traefik` recipe via abra in **wildcard / file-provider mode**, for
  end-to-end fidelity (canonical `web`/`web-secure` entrypoints + proxy/swarm conventions every
  recipe expects — this also fixed an entrypoint-name mismatch the custom build hit). NO ACME, NO
  DNS token on the box:
  - `WILDCARDS_ENABLED=1` + append `compose.wildcard.yml`; the pre-issued cert is fed as the
    `ssl_cert`/`ssl_key` swarm secrets (v1) via `abra app secret insert … -f` from
    `/var/lib/ci-certs/live/{fullchain,privkey}.pem`. The file provider serves it (`tls.certificates`).
  - `LETS_ENCRYPT_ENV=` **empty** on the traefik app *and* on every test app → the recipe's
    `tls.certresolver=${LETS_ENCRYPT_ENV}` label resolves to no resolver → routers serve the
    wildcard via SNI from the file provider, ACME never fires. (Verified: 0 ACME log lines.)
  - Reproducibility (D8): `scripts/deploy-proxy.sh` is idempotent (ensures local abra server, fetches
    recipe, writes the wildcard/no-ACME env, inserts cert secrets, deploys). Documented in
    `docs/install.md`. The custom `modules/traefik.nix` was removed; `modules/swarm.nix` keeps swarm
    init + `proxy` net + firewall 80/443.
  - **Renewal (manual, ~90d):** operator re-issues the wildcard at the same paths, then
    `abra app secret rm traefik.ci.commoninternet.net ssl_cert -n` + re-insert at a new version (bump
    `SECRET_WILDCARD_CERT_VERSION`) and redeploy. (Documented in docs/secrets.md at M7.)
  - **abra teardown syntax** (for harness, §4.3): `abra app undeploy <d> -n`,
    `abra app volume remove <d> -f -n`, `abra app secret remove <d> --all -n`. None take `--chaos`.

- **Infra bring-up = idempotent-reconcile systemd oneshots — SETTLED (M2, orchestrator steer
  2026-05-26).** Every piece of swarm infra that abra deploys (traefik `modules/proxy.nix`, Drone
  `modules/drone.nix`, later comment-bridge + dashboard) is a `systemd.services.<x>` with
  `Type=oneshot` + `RemainAfterExit`, `after`/`requires` swarm-init + docker, `wants`
  network-online, `wantedBy` multi-user, embedding its script via **`pkgs.writeShellApplication`**
  (self-contained in the store, not a `/root/cc-ci` path). The script **reconciles** (inspect →
  converge → no-op if correct) on *every* activation/boot — **no run-once sentinel** — so it
  self-heals drift (stack gone → redeploy; secret missing → re-insert). Fails visibly (failed unit)
  on missing preconditions (e.g. cert absent). Result: a from-scratch install (D8) collapses to
  `git clone` + `nixos-rebuild switch` + operator preconditions, no manual post-steps. The old
  `scripts/deploy-*.sh` were folded into these modules and removed. `pkgs.abra` is provided via an
  overlay (`modules/packages.nix`) so all modules share the one pinned build.
  - *Cert rotation note:* the proxy reconcile inserts ssl_cert/ssl_key only if absent; rotating the
    wildcard means bumping `SECRET_WILDCARD_*_VERSION` (operator) so the next reconcile re-inserts.
    Documented in docs/secrets.md at M7.

- **Trigger: POLLING primary, webhook optional — SETTLED (orchestrator design change 2026-05-27,
  supersedes the earlier "keep webhook, do NOT pivot to polling" steer).** Hard constraint: the
  bot/server runs at **READ level, never repo-admin**, and **never self-registers a webhook**.
  - **Polling is PRIMARY and the source of truth for D1.** The bridge polls each enrolled repo's
    open PRs for new `!testme` comments every `POLL_INTERVAL` (30s ≤ 60s). Outbound
    (cc-ci → git.autonomic.zone, the reliably-working direction), needs only read+comment. On
    startup the first poll marks pre-existing comments seen so it doesn't fire on old comments.
  - **Webhook is an OPTIONAL push optimization.** The `/hook` endpoint stays live (HMAC-verified)
    so an *admin-registered* `issue_comment` webhook lowers latency, but the bridge never registers
    one. Manual registration is documented in `docs/enroll-recipe.md`. Both paths share an
    in-memory seen-set keyed by comment id → a comment seen by both fires at most once.
  - **Commenter authorization via org membership (read-level, no admin).** Allowed iff
    `GET /orgs/{owner}/members/{user}` → 204 (verified 2026-05-27: admits bot/trav/notplants, 404
    for a non-member, works with bot read-level basic-auth) **or** the user is in the optional
    `AUTH_ALLOWLIST`. Replaces the earlier `/collaborators/{user}/permission` check, which needs
    repo-admin. Fail-closed on any error.
  - **Enrollment** = add the repo to the bridge `POLL_REPOS` csv + ensure `tests/<recipe>/` exists.
    No webhook required for CI to work. (Why root cause of the old webhook non-delivery doesn't
    matter: polling makes it irrelevant; the operator was whitelisting `ci.commoninternet.net` in
    Gitea's `ALLOWED_HOST_LIST`, but D1 no longer depends on that.)

- **Resource safety: bound live test apps — SETTLED (orchestrator design change 2026-05-27,
  plan §4.2/§4.3).** Do NOT keep multiple test apps deployed at once. Three layers, all configurable:
  - **MAX_TESTS = `DRONE_RUNNER_CAPACITY` = 1** (`modules/drone-runner.nix`, `maxTests` let-binding).
    Drone runs at most MAX_TESTS builds at once and **auto-queues the rest in its native pending
    queue** — no custom queue. Kept at 1 (single 28GiB node, heavy recipes). At capacity=1 there is
    never a concurrent in-flight run, so the bound "at most 1 test app live" holds exactly.
  - **Per-build TIMEOUT = 60 min** (`modules/drone.nix`, `buildTimeoutMinutes`; reconciled
    best-effort via `PATCH /api/repos/recipe-maintainers/cc-ci {"timeout":60}` using the bridge's
    Drone admin token, local `--resolve`, non-fatal). A build over the limit is cancelled by Drone →
    the exec runner kills it → the MAX_TESTS slot frees → the queue advances. Satisfies "continue
    once a test finishes OR times out".
  - **Teardown + janitor backstop.** Each build deploys → runs the 3 stages → undeploys
    (guaranteed `try/finally` in `conftest`/orchestrator). A SIGKILL'd/timed-out build can't run its
    own teardown, so the **run-start janitor** (`lifecycle.janitor`, called before every deploy in
    both fixtures + `run_recipe_ci`) reaps orphaned run apps as the backstop. At capacity=1 the CI
    path will set `CCCI_JANITOR_MAX_AGE=0` (reap any orphan immediately — safe with no concurrent
    runs) in the recipe-CI Drone pipeline; with capacity>1 the janitor MUST stay age-based (default
    2h) to avoid reaping a live concurrent run. Net: at most MAX_TESTS apps ever live.
  - Optional `concurrency: {limit: 1}` in the recipe-CI `.drone.yml` is a redundant belt — primary
    mechanism is `DRONE_RUNNER_CAPACITY`. (Wired when the recipe-CI pipeline lands — see backlog.)

- **D10 recipe #6: bluesky-pds (TLS-passthrough) SWAPPED → n8n — SETTLED (2026-05-27, plan §4.0
  sanctions this swap-with-reason).** bluesky-pds routes via a Traefik **TCP router with
  `tls.passthrough=true`** to an in-container **caddy** that terminates TLS itself and obtains its own
  cert via **ACME**. cc-ci's design is the opposite: the operator gateway passes wildcard TLS through
  to cc-ci's Traefik, which **terminates** it with the pre-issued static wildcard cert, and **ACME is
  hard-forbidden** for commoninternet.net (no DNS token on the box — §4.0/§9). Serving bluesky-pds
  would require either (a) ACME inside caddy (forbidden), or (b) injecting the wildcard cert into
  caddy + a per-host TCP-passthrough router on cc-ci Traefik (recipe-internal surgery + a bespoke
  proxy mode — not a clean shared-harness absorb). This is a genuine design conflict, not a harness
  gap. Per the plan's explicit allowance, **bluesky-pds is a documented non-CI'd recipe** (reason
  here), and **n8n** takes the 6th slot. The 5 required D10 categories are already covered by recipes
  1–5 (simple=custom-html, single-DB+SSO=keycloak, stateful/no-DB=cryptpad, DB+media/large-volume=
  matrix-synapse, multi-service+S3/object-storage=lasuite-docs); n8n adds a 6th real deployable app
  (workflow automation) behind the normal terminate-at-Traefik path.

- **Docker Hub rate limit + mid-breadth prune — FINDING (2026-05-27).** D10 real-`!testme` breadth
  runs exhausted Docker Hub's anonymous pull rate limit (lasuite-docs, 9 images, upgrade stage:
  `toomanyrequests`). Two lessons: (1) **registry pull creds are an A1 operator input** needed for
  reliable heavy-recipe deploys under load (request + sops-store + wire into docker daemon). (2)
  **Don't `docker image prune -af` mid-breadth** — it evicts cached recipe images and forces re-pulls
  that hit the limit. The first lasuite failure was disk pressure (90% full); pruning fixed disk but
  triggered re-pulls → rate limit. Better: rely on the daily autoprune, prune only `dangling`
  (not `-a`) between runs, or grow disk so heavy images stay cached. Net for D10: 5/6 recipes green
  via real !testme; lasuite-docs gated on the rate limit (transient ~hours; durable fix = creds).

## Open (defaults from §8, to confirm as reality lands)

- **Deploy mechanism — SETTLED (M0):** `nixos-rebuild switch --flake /root/cc-ci#cc-ci` run *on
  cc-ci itself*, with the repo materialised on the host at `/root/cc-ci`. Chosen over
  `--target-host`/deploy-rs to avoid pushing large closures over the userspace-tailscaled SOCKS
  proxy (slow/fragile). Atomic rollback preserved by Nix generations (`nixos-rebuild --rollback`).
  The switch is launched as a **detached transient systemd unit** (`systemd-run --unit=ccci-rebuild
  --collect`) so it survives a momentary ssh-over-tailscale drop during activation. For the build
  loop the host copy is synced from the sandbox clone via `tar | ssh` (rsync absent on host);
  source of truth stays the git repo. D8/install.md will document the from-scratch path (clone repo
  on a fresh host, then `nixos-rebuild switch --flake .#cc-ci`).
  - **nixpkgs pin:** flake pins the exact rev cc-ci already ran (`50ab793…`) so the first rebuild
    is a true no-op-then-base. Bump deliberately, never drift.
- **Webhook scope:** default per-repo via enroll script.
- **CI engine: Drone (per plan) — kept, with a noted risk.** nixpkgs 24.11 has Drone **server**
  2.24.0 but `drone-runner-exec` is **abandoned (unstable-2020-04-19)** — the only exec runner Drone
  ever shipped (upstream archived ~2021). The maintained fork **Woodpecker** (2.7.3, with NixOS
  modules) is the alternative. Decision: honor the plan (Drone) because the plan is Drone-specific
  (D7 "Drone's native UI", comment-bridge → Drone API). The 2020 exec runner pairs fine with modern
  Drone server (RPC protocol stable). **Fallback:** if the exec runner proves incompatible/broken,
  pivot to Woodpecker (coop-cloud ships a `woodpecker` recipe too) and record it — like the traefik
  pivot. Re-evaluate at the M2 gate.
- **Drone deployment shape — SETTLED (M2):** mirror the traefik pattern. The **server** is the
  coop-cloud `drone` recipe (drone/drone:2.26.0) deployed via abra (swarm-native, auto-routed by
  traefik at `drone.ci.commoninternet.net`, `LETS_ENCRYPT_ENV` empty → wildcard cert, no ACME),
  with Gitea SSO (`compose.gitea.yml`). The **exec runner** runs as a Nix systemd service on the
  host (`modules/drone-runner.nix`) so it can drive host abra/swarm (plan §4.2). One generated
  `DRONE_RPC_SECRET` is shared: inserted as the server's `rpc_secret` swarm secret AND read by the
  runner from sops. Reproducible deploy: `scripts/deploy-drone.sh`.
  - Gitea OAuth app `cc-ci-drone` created under the bot (client_id `ab4cdb9d-ee96-4867-875f-
    87384505fc52`, redirect `https://drone.ci.commoninternet.net/login`); client_secret +
    rpc_secret stored sops-encrypted in `secrets/secrets.yaml` (A2 internal secrets).
- **Drone runner type:** exec (must drive host abra).
- **Secret tool — SETTLED (M0):** sops-nix. cc-ci decrypts at activation using its **ed25519 SSH
  host key** as the age identity (`sops.age.sshKeyPaths`), so no extra key file to manage on the box.
  Recipients in `/.sops.yaml`: the host age key (`age1h90ut…`, from ssh-to-age) + an off-box
  **master recovery key** (`age1cmk26t…`; private half only at `/srv/cc-ci/.sops/master-age.txt` on
  the build host, never in the repo) for re-keying if cc-ci is lost. Encrypt new secrets by writing
  plaintext into `secrets/<f>.yaml` then `sops -e -i` (run inside the repo so `.sops.yaml` is found).
- **D10 recipe set:** lock six early. Candidates favouring already-mirrored: custom-html (simple),
  cryptpad (stateful no-DB), keycloak (SSO/DB), matrix-synapse (DB+media), lasuite-docs (multi+S3),
  bluesky-pds (TLS-passthrough) — covers all five categories. Confirm during M4–M6.5.

- **Per-run app domain scheme — adapted (M4, deviates from plan §4.0).** Plan §4.0 wanted
  `<recipe>-pr<n>-<short-sha>.ci.commoninternet.net`, but Docker swarm config/secret names
  (`<stackname>_<resource>_<version>`) must be ≤ 64 chars and abra derives `<stackname>` from the
  domain (dots→`_`, hyphens kept). `.ci.commoninternet.net` alone is 22 chars, so long recipe names
  + config names overflow 64 (hit with `custom-html-pr0-m4demo…_nginx_default_conf_v6` = 66). New
  scheme: **`<recipe[:4]>-<6hex(recipe|pr|ref)>.ci.commoninternet.net`** (e.g. `cust-e084bd`) — short,
  unique per run, collision-safe across recipes (full recipe in the hash). Human-readable recipe/PR/
  ref context lives in the Drone build params + the PR comment, not the (ephemeral) domain.

- **abra recipe checkout is volatile — harness uses chaos+offline + a tests/ snapshot (M6).** Many
  abra commands (`app ls`, `secret generate` without flags, version resolution) silently
  `git checkout <version-tag>` in `~/.abra/recipes/<recipe>`, discarding a PR branch's files. To
  test the *PR head code* (not a re-resolved tag): (1) `fetch_recipe` clones the mirror branch/ref
  (private → bot token via per-command `http.extraHeader`, never persisted/logged); (2) all harness
  abra calls that touch the recipe pass `-C` (chaos: use current checkout) `-o` (offline: no remote
  fetch); (3) recipe-shipped `tests/` (D4) are **snapshotted to a temp dir right after fetch**, since
  later abra commands still reset the checkout — the recipe-local stage runs from the snapshot.

## Risks

- **Disk — RESOLVED 2026-05-26.** Original 8.9 GiB root had only ~3.8 GiB free *and* a hard
  **inode** ceiling (586k total, ~6k free) — the flake's nixpkgs fetch (~50k files) hit ENOSPC on
  inodes before bytes. Operator grew the VM to **28 GiB** (22 GiB free, 1.78M inodes / 1.21M free);
  the ext4 fs auto-resized (new block groups carry proportional inodes). Keep aggressive teardown +
  periodic `docker image prune` to avoid regressing during M6.5 breadth.

## Phase 5 / §4 weekly cron (installed 2026-06-01)

**Schedule:** weekly Monday 23:04 UTC (`4 23 * * 1`). First fire T0 = 2026-06-01T23:04Z.

**Mechanism chosen: busybox crond in a persistent tmux session (`cc-ci-crond`).**
- Rationale: NixOS orchestrator VM has no user crontab (busybox crontab requires suid), no user systemd session (no `/run/user/1000`), and `/etc/nixos` is root-only. Busybox crond runs without suid in foreground mode under tmux, survives as long as the orchestrator is up.
- **Boot persistence gap:** if the orchestrator reboots, the `cc-ci-crond` tmux session does not auto-restart. The NixOS fix is to add `services.cron.systemCronJobs` to `/etc/nixos/configuration.nix` (requires root). Current operator workaround: restart tmux session manually after reboot with `CROND=/nix/store/snjjpdgph0hyha4vm58jyk4mpw03wgq3-busybox-1.36.1/bin/crond && nohup $CROND -f -d 5 -c /home/loops/.cc-ci-crontabs >> /srv/cc-ci/.cc-ci-logs/crond.log 2>&1 &`
- Crontab file: `/home/loops/.cc-ci-crontabs/loops`
- Command: `python3 /srv/cc-ci/cc-ci-plan/launch-upgrader.py start` (creates cc-ci-upgrader tmux session)
- Logs: `/srv/cc-ci/.cc-ci-logs/upgrader-cron.log` (crond execution log), `/srv/cc-ci/.cc-ci-logs/crond.log` (crond daemon log)
- Pre-check: `HOME=/home/loops PATH=/home/loops/.local/bin:/run/current-system/sw/bin python3 /srv/cc-ci/cc-ci-plan/launch-upgrader.py status` → returned "stopped" (working environment) ✓

**V8a gap noted:** cc-ci-upgrader session self-terminates after run completion (Claude exits, tmux session closes). Plan requires "stays idle (does NOT self-terminate)." For weekly cron automation the behavior is correct (fresh start on each invocation). Operator UX gap: run summary not viewable at claude.ai/code after completion; summary is written to disk (`/srv/cc-ci/.cc-ci-logs/upgrades/upgrade-all-*.md`). Not fixed; tracked as known gap.

**T0 fire verification:** PASS — T0 fired 23:04Z, Adversary-verified §4 cron PASS @23:20Z (build complete).

**⚠️ SUPERSEDED 2026-06-02 — mechanism migrated to a NixOS systemd timer.** The CronCreate / busybox
approaches above are both retired. The weekly upgrade now runs via a reboot-safe systemd timer
(`cc-ci-upgrade-all.{service,timer}`) declared in the orchestrator flake
(`nix/hosts/cc-ci-orchestrator-hetzner/configuration.nix`), **OnCalendar=Sun *-*-* 02:00:00 UTC,
Persistent=true** (operator moved the schedule from Mon 23:04 → Sun 02:00 UTC). It runs
`launch-upgrader.py start` → `/upgrade-all` DEFAULT, timer-triggered only. This closes the boot/
restart-durability gap noted above (the CronCreate job was in-memory/session-scoped and evaporated
when the Builder session ended at sequence-complete). Next run: Sun 2026-06-07 02:00 UTC.

## Dead-ends
- (none yet)

## Phase 1c (full reproducibility + genuine D8 live rebuild) — 2026-05-27

- **Secrets linkage = git SUBMODULE (deviates from plan §7 flake-input default).** `cc-ci-secrets`
  is mounted as a submodule at `cc-ci/secrets/` rather than a flake `inputs.secrets`. Rationale: a
  private flake input must be re-fetched at **every nix eval**, requiring the bot token persistently
  in nix config/netrc on cc-ci AND the throwaway VM (a token in the store/config = a 2nd out-of-band
  secret, which 1c forbids). A submodule makes `secrets/secrets.yaml` a plain path in the working
  tree → `defaultSopsFile = ../secrets/secrets.yaml` is unchanged (minimal diff, trivially
  byte-identical), and the only credential use is the one `git clone --recursive` at provisioning
  ("the two repos are *given*", Mission §1). Build invocation becomes
  `nixos-rebuild switch --flake 'git+file:///root/cc-ci?submodules=1#cc-ci'` so the submodule tree is
  included. (Revisit if `?submodules=1` proves unreliable on cc-ci's nix version.)
- **Bootstrap key for the throwaway VM = the existing RECOVERY (master) age key, via
  `sops.age.keyFile`.** The recovery key (`age1cmk26…`, private at `/srv/cc-ci/.sops/master-age.txt`)
  is already a sops recipient, so a fresh host with a *different* ssh host key still decrypts every
  secret with no re-keying — this is exactly the §0 argument that defeats "host-key binding".
  Provisioned to the VM at a fixed path (the ONE out-of-band secret). cc-ci itself keeps decrypting
  via its host key (`age.sshKeyPaths`); secrets.nix will offer both identity sources. (Per-host
  re-encrypt is cleaner for a *permanent* new instance — documented as the alternative, not used for
  the throwaway test.)
- **Cert into git:** wildcard cert+key become sops secrets in `cc-ci-secrets`, decrypted at
  activation back to `/var/lib/ci-certs/live/{fullchain.pem,privkey.pem}` via
  `sops.secrets.<name>.path`; proxy.nix keeps reading that path (now sops-sourced, not operator-drop).
- **cc-nix-test final sizing (C6) — SETTLED by operator 2026-05-27: PROMOTE the rebuilt VM.** The
  freshly-rebuilt reproducible VM (the FINAL W5/C4-C5 clean-room throwaway) becomes the canonical
  cc-nix-test; the operator will repurpose it for a live real-traffic test through the public gateway.
- **C6 teardown OVERRIDE (operator, 2026-05-27):** do NOT destroy the FINAL throwaway VM after
  W5/C4-C5 PASSes — keep it RUNNING; defer its C6 teardown until the operator explicitly says
  otherwise. This overrides the plan §5/§6 "destroy the throwaway" for that one VM only. All other
  cleanup proceeds normally (the Builder's first throwaway was already destroyed; RAM accounting holds).

## Phase 1b — lint/format tooling (open decisions §6, settled W0)
- **Formatters/linters (RL1):** Nix = `nixpkgs-fmt` (format) + `statix` (lints) + `deadnix` (dead
  code); Python = `ruff` (lint + format); Shell = `shellcheck` + `shfmt -i 2 -ci`; YAML = `yamllint`.
  Kept `nixpkgs-fmt` over `alejandra` because it was already the repo `formatter` and devshell tool
  (no extra churn / restyle of every .nix). All built from the already-pinned nixpkgs via a flake
  `lint` devshell (`nix develop .#lint`) so CI and local use byte-identical tool versions.
- **Lint entrypoint:** `scripts/lint.sh` (check-only by default; `--fix` auto-applies). The
  `.drone.yml` push pipeline runs it via `nix develop .#lint --command bash scripts/lint.sh`.
- **ruff strictness:** `select = [E,F,W,I,UP,B,C4,SIM]`, `ignore = [E501]` (line length is the
  formatter's job; only un-splittable strings would trip it). `line-length=100`, `target=py311`.
- **Drone lint stage = FAIL (not warn).** The codebase is green now, so enforce from here on — an
  unclean commit fails the `lint` step. (Resolves the §6 open question.)
- **Python type-checking (mypy/pyright): DEFERRED to IDEAS**, not added in 1b. The harness is small
  and dynamically typed around `abra`/subprocess JSON; gradual typing is a larger effort than this
  bounded pass warrants. Revisit if Phase 2's 18-recipe ramp shows type bugs.
- **blocking vs advisory split (§3):** treated as in the phase plan — tests-real, Nix-idempotent,
  no-footguns, no-secrets, log-redaction, harness-DRY = blocking; readability/docs/arch-drift =
  advisory unless a real plan deviation. Recorded per-finding in REVIEW-1b / BACKLOG-1b.
- **cc-ci self-CI push trigger:** the lint stage lives in the `event: push` pipeline. The Gitea→Drone
  push webhook on this instance is flaky (`last_status: None`; documented §4.1) and predates 1b —
  recipe CI uses polling as primary, but cc-ci's *own* self-test/lint relies on the push webhook.
  The lint stage is correctly wired and proven green via the identical `nix develop .#lint` command;
  reliably auto-firing it on every push is tracked as a (pre-existing) infra item, not a 1b lint gap.

## Phase 1b — repo layout (operator review items RL5/RL6, plan §7)
- **RL5 — all Nix code under `nix/`.** Moved `modules/`→`nix/modules/` and `hosts/`→`nix/hosts/`.
  `flake.nix`/`flake.lock` STAY at the repo root (entry point) so the build ref `#cc-ci` and
  `nixos-rebuild --flake '…#cc-ci'` are unchanged — only `flake.nix`'s internal
  `./hosts/cc-ci/configuration.nix` → `./nix/hosts/cc-ci/configuration.nix` changed. Root-relative
  refs inside the moved modules were re-based `../X` → `../../X` (secrets.nix → `../../secrets/`,
  bridge.nix → `../../bridge/`, dashboard.nix → `../../dashboard/`); `configuration.nix`'s
  `../../modules/*` imports are unchanged (both dirs moved under `nix/`, so the relative path still
  resolves). **Toplevel is byte-identical (`8i3jcad9…`) before/after the move** — store derivations
  are content-addressed on the copied file *contents*, and the module `.nix` files aren't part of the
  runtime closure, so relocating folders doesn't change the build. (The operator anticipated a hash
  change; in practice it's stable, which is even stronger for reproducibility.) Living docs
  (README, architecture/install/secrets/enroll) + the `.drone.yml` comment updated to `nix/…`;
  append-only history logs left as the record of what was true then.
- **RL6 — protocol files → `machine-docs/`: DEFERRED to the coordinated end of 1b.** Will `git mv`
  `STATUS*/REVIEW*/JOURNAL*/BACKLOG*/DECISIONS.md` into `machine-docs/` (README.md STAYS at root —
  operator decision, it's the human readme, not a protocol file). The live watchdog (`launch.sh`)
  reads `STATUS-<id>.md`/`REVIEW-<id>.md` at the repo root for handoffs/transition, so this is done
  LAST, in lockstep with the orchestrator updating `launch.sh` + restarting the watchdog — not
  unilaterally and not while a phase transition is pending. The Adversary likewise `git mv`s its own
  REVIEW files at the cutover (single-writer rule).

## Phase 1b — recorded deviation: no `tests/_template/` dir (enroll = copy an existing recipe)
Plan §3's repo layout lists a `tests/_template/` "copy-to-add-a-recipe" dir. It was **never created**
(pre-1b; not introduced or removed by 1b) — instead the documented enroll flow in
`docs/enroll-recipe.md` is **"copy an existing recipe's tree, e.g. `tests/custom-html/…`, then adjust
`recipe_meta.py` + the per-recipe test files."** This satisfies D5's "small, repeatable, documented
operation with no harness surgery" the same way (a concrete recipe is a better starting template than
an abstract skeleton that can drift). Recording per the Adversary's RL3 D5 advisory; not a blocker.

## Phase 1d — generic test suite + layered overlays (design, 2026-05-27)

SSOT: `cc-ci-plan/plan-phase1d-generic-test-suite.md`. Resolves the §6 open decisions.

- **Tier model & op/assertion split (the core call).** A run is a sequence of TIERS — install,
  upgrade, backup, restore, custom — each = `generic default [overridden by a recipe overlay]`. The
  **lifecycle OP** (deploy, upgrade, backup, restore) is owned by the **shared harness**
  (`harness.generic` helpers), NOT duplicated in every test file. A tier's **test file** (generic or
  overlay) carries the ASSERTIONS and calls the shared op helper. This keeps the op single-sourced
  (DRY, DG7) and makes deploy-once trivial: only the orchestrator deploys/tears-down.

- **Override (not additive) — Builder's call (plan §6, operator leaned override).** For each
  lifecycle op exactly ONE assertion file runs, by precedence:
  **repo-local `tests/test_<op>.py` > cc-ci `tests/<recipe>/test_<op>.py` > generic
  (`tests/_generic/test_<op>.py`)**. A present overlay REPLACES the generic for that op. **Invariant:
  no overlay for an op ⇒ the generic runs** (so any recipe is testable with zero config). Repo-local
  wins same-name collisions (upstream is authoritative, plan §2.5); cc-ci's overlay is the curated
  fallback until upstream adopts it. **Extend-by-composition:** an overlay may
  `from harness import generic` and call `generic.assert_serving(...)` / `generic.do_upgrade(...)`
  then add its own assertions — so "extend" needs no separate mechanism.

- **Custom (non-lifecycle) `test_*.py`:** ALL discovered from BOTH locations run additively, opt-in
  (no override, no generic equivalent) — e.g. `test_sso.py`.

- **Deploy ONCE, mutate in place (operator requirement, DG4.1).** The orchestrator deploys the app
  ONCE, runs all tiers against that single live deployment (install asserts; upgrade does
  `abra app upgrade` in place; backup/restore mutate in place; custom asserts), then ONE teardown in
  `finally`. No per-tier/per-overlay `abra app new/deploy/undeploy`. A `CCCI_DEPLOY_COUNT` counter in
  `lifecycle.deploy_app` is asserted == 1 per run (DG4.1 evidence).

- **Deployment-sharing scope & base version (§6 open).** One deployment for the whole lifecycle.
  Base version deployed once = the **previous published version** when an upgrade tier will run and a
  previous exists (so upgrade goes previous→target in place), **else the target** (current/$REF).
  Recipe with only one published version ⇒ upgrade tier is a clean **SKIP** (nothing to upgrade
  from). Standalone generic-install demo (no PR) deploys current.

- **Fail handling across shared tiers (§6 open):** install failing (app never serves) **fail-fasts**
  the run (later tiers can't meaningfully run on a dead deployment) and they report **error/skip**;
  upgrade/backup/restore failures are recorded per-op but do not abort the remaining independent
  tiers where they can still run. Teardown always runs.

- **Backup-capability detection (DG3, §6 open):** auto — scan the recipe's `compose*.yml` for a
  `backupbot.backup` label (verified present in custom-html). `recipe_meta.BACKUP_CAPABLE` (bool)
  overrides the auto-detect. Not capable ⇒ backup+restore tiers are **N/A (skip)**, not failures.

- **Custom install-steps hook (DG5, §6 open):** a shell hook — `tests/<recipe>/install_steps.sh`
  (cc-ci) or repo-local `tests/install_steps.sh` — run by the orchestrator during the install tier
  AFTER `abra app new` + env defaults but BEFORE `abra app deploy`, with env `CCCI_APP_DOMAIN`,
  `CCCI_RECIPE`, `CCCI_APP_ENV` (path to the app .env). Chosen over a fixture/declarative field as the
  simplest thing the harness runs uniformly (can `abra app secret insert`, set env, seed). Graceful
  rule: a recipe with NO hook still attempts the generic install; if it genuinely needs a step it
  FAILS the generic install (reported per-op) — that is correct, not a harness bug.

- **Per-op result vocabulary (Phase-3 feed):** `pass | fail | skip(N/A) | error`. The orchestrator
  prints a per-op summary line per run (feeds DG6 + Phase-3 level).

- **Discovery layout:** cc-ci overlays/custom/hook live in `tests/<recipe>/`; repo-local in the
  recipe repo's `tests/` (snapshotted after fetch, per the existing volatile-checkout handling).
  Generic tier files live in `tests/_generic/` (assertion-only, use the shared live-deployment
  fixtures).

---

## Phase 1e — generic-harness corrections (HC1–HC4)

Three operator-review corrections to the Phase-1d shared harness, settled here (plan §5).

- **HC2 — repo-local approval allowlist (form/location + workflow).** PR-author-controlled code
  (`install_steps.sh`, repo-local `test_*.py`) runs on the CI host with `/run/secrets/*` present, so it
  is **default-deny**. Allowlist file: **`tests/repo-local-approved.txt`** (checked into the cc-ci
  repo, git-auditable). Format: one recipe name per line; `#` comments + blank lines ignored; a lone
  `*` is NOT a wildcard (no global opt-in — every recipe is explicit). **Default: empty ⇒ no recipe
  trusts repo-local code.** Discovery (`resolve_op`/`custom_tests`/`install_steps`) consults the
  repo-local source **only** when `repo_local_approved(recipe)` is true; otherwise precedence is
  **cc-ci > generic** only and repo-local is discovered-but-not-executed. **Workflow:** a cc-ci
  maintainer reviews a recipe's repo-local tests, then adds the recipe name to
  `tests/repo-local-approved.txt` in a cc-ci PR — a deliberate, reviewable act. The gate is centralized
  in `discovery.py` (one reader) so the unit tests pin it.

- **HC3 — generic-by-default opt-out flag (name/granularity + recipe_meta).** Generic assertions run
  **additively** alongside any overlay by default. Opt-out, in increasing specificity (any one skips):
  env **`CCCI_SKIP_GENERIC`** (truthy ⇒ skip generic for ALL ops), env
  **`CCCI_SKIP_GENERIC_<OP>`** (e.g. `CCCI_SKIP_GENERIC_UPGRADE` ⇒ skip generic for that op only), and
  declarative **`recipe_meta.SKIP_GENERIC`** = a list of op names (or `["all"]`) so the opt-out is
  per-recipe and visible in git, not a hidden global. Truthy = `1/true/yes/on` (case-insensitive).
  **Op-vs-assertion split:** a mutating op (upgrade/backup/restore) is performed **once by the
  orchestrator** (the harness owns the op); then the generic assertion file (unless opted out) and the
  overlay assertion file both evaluate the **shared post-op state**. Op results that an assertion needs
  (pre-upgrade identity, backup snapshot_id) are passed op→assertions via a run-scoped JSON state file
  at `$CCCI_OP_STATE_FILE` (read by `harness.generic.op_state()`); never logged. Overlays that need to
  **seed pre-op state** (data-continuity markers, the backup→restore mutation) ship an optional
  `tests/<recipe>/ops.py` with `pre_install/pre_upgrade/pre_backup/pre_restore(domain, meta)` callables
  the orchestrator runs **before** the op (repo-local `ops.py` is allowlist-gated like other repo-local
  code). Overlay `test_<op>.py` files are now **assertion-only** (they no longer call `generic.do_*`).

- **HC1 — DG4.1 deploy-count vs the in-place chaos upgrade.** The upgrade tier now upgrades to the
  **PR head** (code under test), not a published tag: deploy the previous published version (base),
  **re-checkout the PR head** (recorded as the recipe repo HEAD right after fetch, before any
  version-tag checkout), then **`abra app deploy --chaos`** in place = the upgrade. The deploy-count
  guard counts **`abra app new` installs only** (`_record_deploy()` fires in `deploy_app()`, NOT in the
  chaos redeploy, which calls `abra.deploy` directly) — so a run is still **deploy-count == 1** and the
  legitimate in-place chaos upgrade is not flagged. **Moved assertion (adapted):** prev→PR-head may not
  bump the coop-cloud version label, so `assert_upgraded` accepts ANY of: version-label change, image
  change, or a **chaos label** now present carrying the PR-head commit (a chaos deploy stamps
  `coop-cloud.<stack>.chaos`/`.chaos-version`) — the chaos label IS the proof PR-head was deployed.
  Non-PR `!testme` (no SRC/REF): "PR head" = the catalogue current checkout, so upgrade is prev→current
  — still a genuine move via chaos. (Exact chaos label name verified on the live abra during E2.)

## Phase 2 — per-recipe test authoring (design, 2026-05-28)
Inherits the Phase 1d/1e shared-deployment + additive-overlay + op/assertion-split model. Phase 2
adds **content**, not infra, with a few small harness primitives ported from
`references/recipe-maintainer/utils/tests/helpers.py`.

- **Per-recipe layout (per plan §4.1).** The cc-ci `tests/<recipe>/` dir continues to use the
  Phase-1d/1e overlays at the top level (`test_install.py`, `test_upgrade.py`, `test_backup.py`,
  `test_restore.py`, `ops.py`, `recipe_meta.py`, optional `install_steps.sh`). NEW Phase-2
  subdirectories:
  - `tests/<recipe>/functional/` — parity-port tests (one per recipe-maintainer `tests/*.py`) +
    ≥2 NEW recipe-specific functional tests (P2/P3). Each file is `test_*.py` (pytest-discoverable);
    each parity port carries a **`SOURCE = "recipe-info/<recipe>/tests/<file>"`** comment near the
    top so the audit trail is in the file, not just in PARITY.md.
  - `tests/<recipe>/playwright/` — browser flows (P6) where the app's UX is a UI flow. Same
    `test_*.py` convention; each file imports `playwright.sync_api`.
  - `tests/<recipe>/PARITY.md` — required mapping table (P2) with one row per
    recipe-info parity test: `| recipe-maintainer file | cc-ci file | what's verified | status |`.
    A deliberate non-port is a documented row in DECISIONS.md (linked from PARITY.md), not a silent
    omission.
- **Discovery for the new subdirs.** `runner/harness/discovery.custom_tests` recurses into
  `tests/<recipe>/functional/` and `tests/<recipe>/playwright/` (in addition to the top-level glob),
  so Phase-2 functional tests run as part of the **custom** stage automatically. Repo-local (HC2)
  gate still applies if the recipe is approved; otherwise only cc-ci's own functional/ + playwright/
  run. The top-level `test_install.py`/etc. continue to drive the lifecycle overlays — the
  `functional/` + `playwright/` files are **always custom-stage**, never lifecycle (so they don't
  perform an op; they assert against the post-install live deployment).
- **Vendored helpers in `runner/harness/`.** Capabilities ported from `recipe-maintainer/utils/tests/
  helpers.py` (cc-ci is self-contained at runtime — does NOT import recipe-maintainer's workspace,
  per plan §8 default):
  - `harness.http` — `http_get(url, headers=, timeout=) -> (status, json_or_None)`,
    `http_post(...)`, `retry_http_get(url, timeout=, **)`, `wait_for_http(url, label, max_wait=)`,
    `assert_converges(fn, description, max_wait=, interval=)`. (Several variants exist
    `lifecycle.http_fetch/http_get/http_body` already; the harness.http module is the **canonical**
    Phase-2 HTTP API for tests; lifecycle.* helpers stay for infra-level checks.)
  - `harness.abra_tty` — `script -qefc "abra …" /dev/null` wrapper for the abra commands that
    require a TTY (backup/restore/secret/run/logs/lint), used by parity tests that drive abra
    directly. Lifecycle already exposes typed wrappers — this is for tests that need raw shell-abra.
  - `harness.deps` — dependency resolver primitive. Reads `tests/<recipe>/recipe.toml`
    (`requires` / `test_requires`), deploys each declared dep via the same `lifecycle.deploy_app`
    + `wait_healthy` path (so the dep is a real `<dep[:4]>-<6hex>.ci.commoninternet.net` on the
    same swarm), persists per-run, tears down with the parent in the orchestrator's `finally`.
    Heavy recipes sequence sequentially; `MAX_TESTS`/node budget is the cap.
  - `harness.sso` — OIDC-flow primitive (Q2 deliverable). Given a deployed provider domain and a
    recipe-defined realm/client/test-user, performs the full "deploy provider → setup realm/client
    via admin API → obtain access token (password + client-credentials grants) → assert protected
    API call accepts it" assertion. Reusable by every SSO-dependent recipe (cryptpad, lasuite-*,
    immich, etc.). Setup scripts ported from `recipe-info/<dep>/setup_<provider>_integration.py`.
  - `harness.data_integrity` — backup data-integrity primitive: a recipe-aware "seed a marker
    →  backup → mutate → restore → assert seeded marker survived" helper around `lifecycle.exec_in_app`
    / `http_get` (the recipe chooses the marker mechanism, the helper guarantees the pattern).
- **Run-scoped credentials for SSO/recipe-specific tests** (plan §4.4 class-B). Generated secrets
  (realm/client/test-user passwords, API tokens) persist for the run via the existing
  `runs/<app-name>/` mechanism (Phase 1d). Destroyed at teardown alongside abra secrets/volumes.
- **Recipe-versioned tests (anti-anchoring).** Per plan §7.1, tests read versions/endpoints
  dynamically (the app's own discovery endpoints, env from `live_app`) — never hardcode published
  release values. Each functional test file declares the recipe-info SOURCE path it ports from so
  the Adversary can audit parity cold.
- **Heavy-recipe parking.** Drone's `MAX_TESTS=1` + per-build timeout already serialize runs; for
  Phase 2 we DO NOT lift it. Within a single run, the orchestrator deploys deps before the
  recipe-under-test sequentially (never concurrently) per plan §4.2.

## Phase 2 Q3.4 — cryptpad create-pad deeper test deferral (2026-05-28)

**Status:** Deferred to Q3.4 follow-up (or Q5 catch-up), with Adversary sign-off pending per
plan §7.1.

**What's deferred:** The "create-an-object + read-it-back" deep test for cryptpad —
authenticate-and-create a real pad in the browser, type a uniquely-marked content string, reload
the page (retaining the client-side encryption key in the URL fragment), assert the marker
survives. This is the canonical create-and-read-back per plan §4.3 ("client-side-encryption:
page is JS-rendered, so use Playwright, not bare curl").

**Why deferred (the technical reason):**
- CryptPad's pad-creation client-side flow is **version-specific**. In the recipe under test
  (10.6.0+5.7.0), visiting `/pad/` does NOT auto-inject a fragment-keyed pad URL; CryptPad
  requires the user to explicitly click a "new rich text" / "new pad" link from the landing
  page, AND those UI selectors (`.cp-apps-grid a`, `[data-app='pad']`, `a[href*='/pad/']`) are
  not stable across CryptPad versions.
- Three attempted drafts during Q3.4 each failed cold on this:
  1. Type + reload + content-survives: contenteditable inside nested iframe with origin
     mismatch (SANDBOX_DOMAIN).
  2. Direct-`/pad/`-then-fragment: no fragment ever appeared on this version.
  3. Click-fallback for known app-launch selectors: none of the candidate selectors matched.

**The maximal testable subset that IS shipped (P3 floor met):**
- `tests/cryptpad/functional/test_health_check.py` — parity HTTP 200.
- `tests/cryptpad/functional/test_spa_assets.py` — CryptPad branding + canonical asset paths
  in served HTML. Catches the wedged-server-fallback-page failure mode.
- `tests/cryptpad/playwright/test_pad_create.py` — Chromium renders the SPA, asserts brand
  + canonical asset references + zero non-filtered JavaScript console errors.

The Playwright test exercises the JS pipeline in a real browser (per §4.3 directive); the
piece NOT exercised is the user-action-driven pad lifecycle. **What's required to lift the
deferral:** pin a specific CryptPad app-launch contract (CryptPad's source has app-launch
URL patterns like `/pad/?new=1` on some versions) OR write a Playwright helper that walks the
SPA's main menu via a stable accessibility tree (role-based selectors instead of CSS).

Adversary may file F2-N requesting full create-pad coverage; the answer above is the
honest technical reason + the maximal subset. Logged here per plan §7.1.


---

## Phase 2 — nested DOMAIN-derived subdomains flattened to single-label wildcard siblings

**Decision (settled):** When an enrolled recipe routes additional services on **nested subdomains
derived from `DOMAIN`** (e.g. lasuite-drive `MINIO_DOMAIN="minio.${DOMAIN}"` +
`COLLABORA_DOMAIN="collabora.${DOMAIN}"`; lasuite-meet `LIVEKIT_DOMAIN="livekit.${DOMAIN}"`), the
recipe's `recipe_meta.EXTRA_ENV(domain)` MUST override those vars to a **single-label sibling under
the wildcard** — `minio-<domain>`, `collabora-<domain>`, `livekit-<domain>` — NOT the recipe's
default `<svc>.<domain>`.

**Why:** cc-ci's TLS cert is the operator's pre-issued wildcard `*.ci.commoninternet.net` (+ bare
`ci.commoninternet.net`) — §4.0/§1.5, renewed out-of-band, no ACME. A wildcard matches exactly **one**
label. The per-run app domain is already one label (`lasuite-drive-pr<n>-<sha>.ci.commoninternet.net`),
so a nested `minio.lasuite-drive-pr<n>-<sha>.ci.commoninternet.net` is a **2-label** name the wildcard
does NOT cover → Traefik would serve an invalid cert on that router and the service is unreachable
over HTTPS. Re-prefixing with a hyphen keeps it one label (`minio-lasuite-drive-pr<n>-<sha>` +
`.ci.commoninternet.net`), covered by the same wildcard, routed by Traefik's swarm provider with **no
cert work and no gateway change** (the gateway already passes the whole wildcard, §4.0). We must NOT
mint per-host certs / ACME for these (class-A1 boundary, §9).

**Scope:** purely a per-recipe `EXTRA_ENV` concern (no shared-harness change). Recipes with no
DOMAIN-derived nested subdomains (most) are unaffected.

## Phase 2 — `services_converged` treats a `replicas: 0` one-shot as converged

**Decision (settled):** `runner/harness/lifecycle.py::services_converged` now considers a service
converged when `cur == want` (desired replica count met), removing the prior
`or want == "0"` rejection.

**Why:** lasuite-drive's `minio-createbuckets` is declared `deploy: {mode: replicated, replicas: 0,
restart_policy: {condition: none}}` — an **on-demand one-shot** (scaled up manually only when buckets
need (re)creating; it `mc mb …` then `exit 0`). `docker stack services` reports it `0/0`. The old
check rejected any `want == "0"` row, so the stack could **never** report converged → every deploy
hung until `deploy_timeout`. A service AT its desired count (including 0/0) is converged; a service
still spinning up shows `0/1` (`cur != want`) and is correctly not-yet-converged, so the HTTP
readiness wait still gates real liveness. Safe for all currently-green recipes (their services are
all N/N with N>0; the `0/0` case did not previously occur). Buckets/migrations that the one-shot
performs are run on-demand in the recipe's `setup_custom_tests.sh` (post-deploy), not relied upon for
generic-install convergence (the SPA at `/` serves 200 without them).

## 2026-05-28 — Docker Hub auth: declarative config.json via sops (rate-limit fix) — SETTLED

**Context.** Heavy Phase-2 recipe deploys exhausted Docker Hub's anonymous pull rate limit
(100/6h per shared IP `68.14.43.142`) → `toomanyrequests` blocked all new deploys. Operator
provided a read-only Docker Hub PAT (Class A1 registry creds, plan §1.5): `DOCKERHUB_USERNAME=nptest2`
+ `DOCKERHUB_TOKEN` in `/srv/cc-ci/.testenv`. Authenticated pulls = 200/6h **per-account**.

**Decision.** Wire it declaratively (survives a 1c NixOS rebuild), not just an imperative login:
- **Secret:** `secrets/secrets.yaml` (cc-ci-secrets submodule, commit `cdd5e0a`) gains key
  `dockerhub_auth` = `base64("nptest2:<PAT>")` — i.e. the exact `auth` field docker config.json
  wants, so the nix template is a pure render (no runtime base64). sops-encrypted to host+master age
  recipients (edited on cc-ci using its ssh-host-key→age identity via `nix shell nixpkgs#sops`;
  plaintext shredded; PAT never committed plaintext nor exposed in process args/logs).
- **Render:** `nix/modules/secrets.nix` adds `sops.secrets.dockerhub_auth` + a
  `sops.templates."docker-config.json"` that renders `/root/.docker/config.json` (0600, root) at
  activation. It becomes a symlink to `/run/secrets/rendered/docker-config.json`.
- **Why /root:** the drone exec runner runs pipelines as `User=root` (drone-runner.nix), and manual
  deploys ssh in as root — so `/root/.docker/config.json` covers both the `!testme` CI path and
  manual ops. Single config, single user.

**Swarm-propagation question — RESOLVED empirically (no `--with-registry-auth` / pre-pull needed).**
The operator/Adversary flagged that a node `docker login` may NOT propagate to swarm SERVICE-task
pulls. Tested on cc-ci with the authenticated config.json in place:
- Account ratelimit baseline 197/200 (source = account hash `b662dd8b-…`, not the IP).
- Deployed **uncached** `n8nio/n8n:2.20.6` via abra (`RECIPE=n8n STAGES=install`). The swarm service
  task pulled it to `1/1 Running` with **no `toomanyrequests`**.
- Account counter dropped 197 → 196 (manager manifest resolution) → **195** (agent layer-manifest
  pull), source still the account hash. So abra's `docker stack deploy` propagates the cred to the
  swarm task pull on this single-node swarm — billed to the account, not the anon IP.
- Corroborating: the earlier lasuite-drive deploy resolved **12** images with no `toomanyrequests`
  while anon budget was ≤4 — impossible anonymously → manager resolution is authenticated too.

So: declarative root `config.json` is sufficient end-to-end here; `--with-registry-auth` is not
required (abra/SDK attaches it). **Caveat (Phase 2b):** 200/6h may still be tight for a full ~18-recipe
sweep; the permanent structural fix is a registry pull-through cache authenticated with this same PAT.

---

## Phase 2w — warm canonical + `--quick` (2026-05-28)

**Stable-domain scheme for warm apps: `warm-<recipe>.ci.commoninternet.net`.** Distinct from cold
per-run `<recipe[:4]>-<6hex>` (naming.app_domain) so a warm app is never confused with a disposable
cold run. Live-warm keycloak = `warm-keycloak.ci.commoninternet.net`; data-warm canonicals (W1) =
`warm-<recipe>...`. Risk to watch: longer stack name vs swarm's 64-char config/secret limit —
verified per-recipe on first deploy; shorten the scheme if any recipe's secret name overflows.

**Realm is the per-run isolation unit on the shared live-warm keycloak (WC1).** Instead of
co-deploying a fresh keycloak per dependent run, dependents use the one live-warm keycloak and create
a **per-run namespaced realm+client+user**, deleted at run teardown. Realm name =
`<parent_recipe>-<6hex>` where 6hex is the parent's per-run domain label suffix — unique per
(parent, pr, ref) so concurrent dependents never collide, and traceable for debugging. (Was
`realm=parent_recipe`, which would collide across concurrent same-recipe runs.)

**Warm keycloak is declarative INFRA, not warm DATA.** The live-warm keycloak service is brought up
by a Nix systemd-oneshot reconciler (converges to deployed+healthy at the stable domain), exactly
like the traefik recipe deploy — so it IS in the D8 reproducibility closure (re-warmable from
scratch) and self-heals on activation/boot. Only warm *volumes/snapshots* (W1+) are cache excluded
from D8. The keycloak's realm data is ephemeral per-run, so nothing persistent to exclude.

**Live-warm is an optimization layer with a cold fallback.** If no warm keycloak is present (e.g. a
from-scratch host before the reconciler has run, or the warm app is down), the keycloak dep path
falls back to the existing cold co-deploy so dependent runs still work. The warm path is preferred
when available.

## Phase 2w — design update: unpinned warm/infra + health-gated rollback (2026-05-28/29)

**Warm/infra apps (traefik + keycloak) auto-update to LATEST nightly, health-gated (operator).**
Supersedes the W0.3 pinned `kcVersion`. Keycloak is now unpinned like traefik: reconciler `abra
recipe fetch` latest + chaos deploy; keep secret-generate-only-if-missing + health-wait. D8 holds
because the recipe is fetched at *activation* (runtime), so the nix store closure is byte-identical
regardless of which keycloak version is live.

**Snapshot helper (WC3) — format + path.** `runner/harness/warmsnap.py`. A snapshot is a **raw tar
of each docker volume belonging to the app's stack**, taken **while the app is undeployed** (nothing
writing → consistent). Stored under `/var/lib/ci-warm/<recipe>/` as `<recipe>.snapshot.tar` + a
`<recipe>.meta.json` (commit/version/timestamp/volume list). **One last-good per app**, replaced
**atomically** (write to `.tmp` then `rename`). Restore: for each volume, clear `_data` and untar
back. Docker volumes are stack-scoped (`<stack>_<vol>`); the helper enumerates them via
`docker volume ls` filtered to the stack. Reused by WC1.1 (pre-upgrade snapshot of keycloak) and WC5
(promote-on-green-cold). Warm snapshots are **cache, excluded from the D8 closure** (WC8).

**Alert mechanism — sentinel files relayed by the Builder loop.** The warm/infra reconciler is an
autonomous bash systemd unit on cc-ci; it cannot call the agent's `PushNotification` tool. So a
reconciler that rolls back (WC1.1) or holds a major/manual-migration upgrade (WC1.2) writes a JSON
**alert sentinel** to `/var/lib/ci-warm/alerts/<ts>-<app>-<reason>.json` (fields: app, reason
[rollback|held-major|held-manual-migration], from_version, to_version, release_notes, ts). The
Builder loop, each wake, scans that dir; for each new alert it (a) issues `PushNotification` to the
operator, (b) records it in STATUS-2w/JOURNAL-2w, (c) archives it to `alerts/seen/`. This bridges the
autonomous reconciler to operator visibility (latency = next Builder wake; acceptable for an alert).

**Re-sequence:** WC1.1's keycloak rollback needs the WC3 snapshot helper, so build that FIRST, then
rewrite the reconciler ONCE into the unpinned + WC1.2-safety-gated + WC1.1-health-gated-rollback form
(avoids reworking the reconciler twice). The W0.3 reconciler is INTERIM until then.

## Phase 2w — W0.6 reconciler: version model + deploy-by-tag (2026-05-29)

**Reconcile entrypoint in Python, packaged in the nix store.** `runner/warm_reconcile.py`, invoked by
the systemd unit as `${pyEnv}/bin/python3 ${../../runner}/warm_reconcile.py <app>` (the runner/ dir is
copied into the store → D8-clean, no dependence on the /root/cc-ci checkout). Reuses
warmsnap/sso/abra/lifecycle so there is ONE snapshot impl (also used by the runner for WC5). Replaces
the bash reconcile in warm-keycloak.nix.

**"latest" = newest published version TAG, deployed pinned (not chaos-of-main).** WC1.2's "major
recipe-version bump" detection needs comparable versions, which chaos (deploy main HEAD) doesn't give.
So the reconciler resolves latest = `git tag | sort -V | tail -1` (valid coop-cloud version tags),
records current = the app .env `VERSION`, and deploys the chosen tag pinned (`abra app deploy <domain>
<version> -o -n -f`, after `git checkout <tag>`). "Auto-update to latest" is satisfied by converging
to the newest tag; "chaos" in the operator note is read as "auto-deploy latest", and tag-pinning is
the correct mechanism for a version-gated auto-update.

**coop-cloud version format is `<recipe-semver>+<app-version>` (observed), not the plan's
`<upstream>+<recipe-semver>`.** Evidence: keycloak `10.7.1+26.6.2` → image `keycloak:26.6.2`; n8n
`3.2.0+2.20.6` → image `n8nio/n8n:2.20.6` (the post-`+` part is the app image tag). So the **recipe
semver is the part BEFORE `+`**. WC1.2's "major recipe bump = breaking" keys off the major (first)
component of the pre-`+` recipe semver (e.g. 3.x→4.0 = held). Secondary signal: scan the target's
`releaseNotes/<version>.md` for manual-migration markers.

**Scope order for W0.6:** keycloak first (the W0 focus, stateful → snapshot path); apply the same
health-gated + safety-gate pattern to traefik (stateless, version-rollback-only) afterward by
migrating proxy.nix onto the shared reconcile entrypoint.

## Phase 2w — W1 canonical registry design (WC2/WC3) (2026-05-29)

**Enrollment is declarative per-recipe via `recipe_meta.WARM_CANONICAL = True`** (consistent with how
DEPS/EXTRA_ENV are declared — enrolling a recipe stays a `tests/<recipe>/` change, D5). A recipe so
flagged gets a DATA-WARM canonical. Prove the model on a couple of recipes (custom-html simplest:
stateful, no external DB), NOT all (the nightly sweep populates the rest over time).

**Stable domain `warm-<recipe>.ci.commoninternet.net`** (already decided for keycloak; same scheme for
canonicals). Distinct from cold `<recipe[:4]>-<6hex>`. Watch the swarm 64-char secret-name limit
per recipe on first deploy.

**Known-good state per canonical, under `/var/lib/ci-warm/<recipe>/`:** `last_good` (version string,
already written by warm_reconcile), `snapshot/` (warmsnap, W0.5), and a small `canonical.json`
registry record `{recipe, domain, version, commit, status, ts}`. The DATA VOLUME is retained while
the app is undeployed (data-warm). These are cache (excluded from D8, WC8).

**Data-warm lifecycle (new `runner/harness/canonical.py`):** `is_enrolled(recipe)` (reads
WARM_CANONICAL), `canonical_domain(recipe)`, `read/write_registry(recipe)`, `deploy_canonical(recipe)`
(deploy `warm-<recipe>` at last_good, reattaching the retained volume → warm boot), `undeploy_keep_
volume(recipe)` (undeploy, volume retained = idle data-warm), `seed_canonical(recipe, version, commit)`
(record + snapshot; the volume becomes the canonical). LIVE-warm (keycloak, always up) vs DATA-warm
(canonicals, undeployed when idle) both use `warm-<recipe>` + warmsnap.

**W1 scope vs W3:** W1 builds the registry + data-warm lifecycle and proves it (seed a custom-html
canonical → undeploy keep volume → redeploy reattach → data survives; re-warmable from scratch).
**Automatic promote-on-green-cold (WC5) + nightly (WC6) are W3** — for W1 the canonical is seeded
programmatically to prove the model; the cold-advances-canonical wiring comes later.

## Phase 2w — W3 WC5 promote-on-green-cold mechanism (2026-05-29)

**Promote = re-seed the canonical from a fresh deploy of the green-verified latest (NOT "keep the
cold run's per-run volume").** Rationale: a cold run uses a fresh per-run domain `<recipe>-<6hex>`
with a fresh volume (cold stays authoritative + fresh); its volume names are per-run-specific and
differ from the canonical's `warm-<recipe>` volume names, so the per-run volume can't be directly
reused as the canonical without a fragile name-remap. AND the cardinal guardrail "never lose the
known-good" forbids touching the existing canonical until a new green one is ready.

So: on a run that is **enrolled (recipe_meta.WARM_CANONICAL) + GREEN + COLD (not --quick) + on LATEST
(no PR head, i.e. REF empty — the nightly/manual-latest run, NOT a PR `!testme`)**, AFTER the normal
per-run teardown, the orchestrator PROMOTES: deploy `warm-<recipe>` at latest → wait healthy →
undeploy → `canonical.seed_canonical(version=latest, commit=head)` (snapshot-while-undeployed +
atomic registry/snapshot replace). The old known-good is replaced ATOMICALLY only on a green promote
(a red run never reaches promote → known-good safe). The canonical's data = a clean install of the
green-verified latest (a valid known-good baseline; --quick reattaches + upgrades it). Cost: one extra
(canonical) deploy per promote — acceptable for cold/nightly (not latency-sensitive). The FIRST such
green run SEEDS the canonical. `--quick` never promotes (proven W2). Only cold advances (WC5).

Promote gate predicate (unit-tested): `is_enrolled(recipe) and overall==0 and not quick and not ref`.
(`not ref` = a catalogue-latest run, i.e. the nightly sweep or a manual `RECIPE=<r>` run — a PR
`!testme` carries REF=PR-head and must NOT advance the canonical to a PR's code.)

## Phase 2 — heavy-recipe upgrade tier disk constraint (28GB host) — SETTLED finding @2026-05-29
The upgrade tier (HC1: prev published → PR-head via in-place `abra app deploy --chaos`) cannot
complete for recipes whose successive releases bump multi-GB image tags, because the rolling update
must hold BOTH versions on disk transiently. Proven on lasuite-drive: onlyoffice 9.2 → 9.3.1.2
(3.94GB each) + collabora two versions → ~10GB office images at once vs ~14GB docker headroom on the
28GB host → 99% → deploy fail. **No harness fix is possible** (the prev images are running, so they
are neither dangling-prunable nor `rmi`-able when the new must be pulled). install/backup/restore/
custom (single version) fit and pass. Resolution = grow the host disk (Class A1 operator input,
DEFERRED.md 2026-05-29). Until then, heavy recipes are verified via their maximal testable subset
(install+backup+restore+custom) with the upgrade tier flagged as a genuine env-level (disk) blocker
per plan §7.1 (Adversary sign-off required). The cleanup runbook for an over-full host: `pkill -f
run_recipe_ci.py`; `docker stack rm <leftover>`; remove its volumes+secrets; `docker image prune -f`.

## SSO-provider policy (operator, 2026-05-29) — keycloak is the DEFAULT; authentik is NOT a DONE gate
Standing policy for all Phase-2 (and later) recipe OIDC/SSO testing:
- **keycloak is the default SSO provider.** Default ALL recipe OIDC tests to keycloak (live-warm WC1).
- **Do NOT test authentik↔keycloak integration**, and do NOT enroll authentik merely to "prove
  pluggability" / second-provider coverage. **Phase-2 DONE is NOT gated on authentik.**
- Enroll authentik + add `setup_authentik_realm` (the provider-pluggable backend in
  `runner/harness/sso.py`) **ONLY if a recipe genuinely REQUIRES authentik** (cannot work under
  keycloak). If it works with keycloak, use keycloak.
- **cryptpad:** its recipe-maintainer upstream SSO test uses authentik, but cc-ci tests cryptpad's OIDC
  under **keycloak** (equally valid). Same for any recipe whose upstream happens to use authentik but
  functions fine under keycloak.
- The OIDC FLOW primitives (`oidc_password_grant`, `assert_discovery_endpoint`) are already
  provider-agnostic; only realm/client SETUP is provider-specific, and we only need the keycloak setup
  (`setup_keycloak_realm`) unless/until a recipe forces authentik.
Consequences: DEFERRED #9 (authentik enrollment) re-entry trigger narrowed to "a recipe requires
authentik"; F2-7 (authentik backend) is not a DONE blocker. plan-sso-dep-testing.md §6 updated by the
orchestrator to match.

## Phase 2pc — image-prune policy; local store IS the cache; registry pull-through DROPPED (2026-05-29) — SETTLED
Decision (PC1): removed `virtualisation.docker.autoPrune` (it ran `docker system prune --force --all
--filter until=24h` daily). The `--all` evicts every image not used by a *running* container —
between runs no test apps run, so it wiped the cached recipe base images → cold re-pull → Docker-Hub
rate-limit churn (JOURNAL-2 507/542/690-693). Replaced with `nix/modules/docker-prune.nix`: the
`ci-docker-prune` daily timer + oneshot, a **surgical triple-gated** prune that no-ops unless ALL of
(1) `/` ≥ 80%, (2) no run-app stack live, (3) no swarm service converging; and when it runs prunes
only **dangling images + stopped containers + dangling build cache, `until=24h`** — never `--all`
(keeps tagged base/in-use images), never `--volumes` (warm canonical data). Teardown
(`lifecycle.teardown_app`) already removes only services/volumes/secrets/.env, never images — kept.
Why: on this **single host Docker's own local image store IS the cache** — a pulled image stays and
redeploys reuse local layers with no re-download (proven: redis:7-alpine cold pull 5303ms w/ 6 layer
downloads → after `service rm` teardown the image is retained → warm redeploy "Image is up to date"
674ms, no bytes); the PAT-authenticated daemon (200/6h) makes the residual warm-deploy manifest check
free of rate-limit pressure. So *keeping* the store recovers ~all the benefit a cache would give.

Decision (registry pull-through cache): **DROPPED here, deferred to IDEAS / Phase 2b** (operator
scope correction 2026-05-29, mid-phase). A `registry:2` pull-through cache's distinctive wins —
multi-node fan-out, surviving prune/VM-rebuild on *separate* storage, cache-miss authentication —
**don't apply** to a single authenticated non-pruning host (one node; co-located cache lost on a
recreate anyway; daemon already authenticated). It would add a registry service + daemon-mirror
config + cache GC for marginal gain. **Revisit ONLY if** (a) cc-ci goes multi-node, OR (b) Phase-2b
measurement shows cold-deploy pull time is a real bottleneck AND the cache can live on
recreate-surviving storage (Incus volume / host b1 path, not the VM's ephemeral disk). No registry
code was written (caught during orientation) — nothing to revert.

---

## 2026-05-29 — Real-abra-only deploys; abra convergence by default; READY_PROBE only when abra doesn't fit (operator principle; plan.md §9)

**Decision (operator, 2026-05-29).** CI deploys/upgrades MUST use **real abra commands** — never
`docker service update`/`docker service scale` to surgically patch a stack into health (that would
test a non-abra path and can mask a broken deploy). **Prefer abra's own convergence checks by
default.** Only skip abra's convergence monitor (`abra app deploy -c/--no-converge-checks`) and
substitute a **harness READY_PROBE** when abra genuinely does not fit — e.g. its convergence window
is too short for a heavy app and it `FATA deploy failed`s on a deploy that DOES converge given time.

**When you do skip abra convergence, the rules are:**
- The deploy stays **real abra** (`abra app deploy [-C] -c`); only abra's *waiting* is replaced, not
  the deploy mechanism. `docker stack deploy` still applies the real spec.
- The harness replacement MUST be a genuinely **STRICT** readiness test: **all swarm services N/N**
  (`lifecycle.wait_healthy` → `services_converged`) **+ a real app-level check** (the app HEALTH_PATH
  AND any recipe `READY_PROBE` — a live HTTP assertion on a real endpoint), bounded by a generous but
  finite deadline (recipe `DEPLOY_TIMEOUT`).
- It MUST **RAISE on actual non-readiness** — never a no-op that lets a failed deploy pass. **Prove it
  has teeth with a negative test.**

**Applied:** F2-12 lasuite-drive upgrade tier. abra's converge monitor FATA'd while the upgraded
collabora `25.04.9.4.1` healthcheck was still in `start_period` (jail/config init), though it
converges via swarm's healthcheck retries. Fix (`e1147b5`): upgrade chaos redeploy uses `abra … -c`;
`generic.perform_upgrade` then owns `lifecycle.wait_healthy` (services N/N + app HEALTH_PATH) +
`lifecycle.wait_ready_probes` (recipe `READY_PROBE` → collabora WOPI `/hosting/discovery` 200),
bounded by `DEPLOY_TIMEOUT`. Teeth proven by `tests/unit/test_f212_upgrade_convergence.py` (`6506c4a`,
5 P7-negative tests: the wait RAISES `TimeoutError` on stuck/never-serving convergence). The lone
`docker service scale …minio-createbuckets` is NOT a bypass — it triggers the recipe's own
`replicas:0` one-shot (Adversary-confirmed). The Adversary still owns confirming "not a weakening" at
the Q3.2 cold-verify.

## 2026-05-29 — READY_PROBE / abra `-c` policy (operator principle; Adversary-recorded)
**Decision (operator, plan.md §9):** deploys/upgrades use REAL abra commands — **never** `docker
service update/scale` to mutate app state. **PREFER abra's own convergence checks by default.** Only
skip abra convergence (`-c`/`--no-converge-checks`) + use a harness `READY_PROBE` when abra genuinely
does not fit — e.g. its window is too short for a heavy app and it FATAs on a deploy that DOES converge
(F2-12: lasuite-drive new collabora 25.04.9.4.1 in healthcheck `start_period`). When skipping:
- the deploy stays **real abra** (only abra's *waiting* is replaced, not the deploy);
- the custom probe MUST be genuinely **STRICT** — all services N/N **plus** a real app-level check —
  and **RAISE** on actual non-readiness; never a no-op that masks a failed deploy;
- prove it has **teeth** with a negative test (cf. F2-12 P7-negative
  `tests/unit/test_f212_upgrade_convergence.py`).
**Adversary status:** the F2-12 lasuite-drive READY_PROBE was cold-verified non-weakening at the Q3.2
re-claim (REVIEW-2 "## Q3.2 … PASS @2026-05-29"): `-c`+owned `wait_healthy`(services N/N + HEALTH_PATH)
+`wait_ready_probes`(collabora WOPI 200) all RAISE on stuck convergence (5 unit tests pass + code-read);
upgrade tier GREEN on the Adversary's own cold run. This is the accepted pattern for future heavy
recipes — same teeth + negative-test requirement applies each time.

---

## 2026-05-29 — R014 lightweight upstream tags → chaos-base deploy (Q3.3 lasuite-meet)

**Problem.** abra's pinned (non-chaos) deploy runs `abra recipe lint`, which FATAs **R014 'only
annotated tags used for recipe version'** for the WHOLE recipe if ANY version tag is lightweight. Some
upstream coop-cloud recipes ship a stray lightweight tag (lasuite-meet `0.3.0+v1.16.0`). This blocked
the upgrade tier's prev-version base deploy.

**Rejected approach (origin-repoint).** Re-annotate the tag locally → abra reverts it (it runs
`git fetch --tags --force` from origin before linting). Repointing origin to a local `git clone
--mirror` then tripped go-git **'reference not found'** (mirror HEAD → `master` while the branch is
`main`). Too fragile; abandoned.

**Decision (chaos-base).** Detect lightweight version tags (`abra.has_lightweight_version_tags`,
read-only). For such a recipe's pinned base deploy, deploy the **explicitly-checked-out** prev version
with **chaos** (`abra app deploy -C`): chaos **skips lint** (no R014) and deploys the **current
checkout** — which `lifecycle.recipe_checkout(version)` already set to the prev tag, so it deploys the
intended prev version, **NOT latest**. (F1d-2's hazard was a *missing* checkout; the explicit checkout
removes it.) **Verified real** by the Q3.3 upgrade crossover `0.2.0+v1.15.0→0.3.0+v1.16.0`. No-op /
stays pinned-non-chaos for all-annotated recipes (most). The deeper fix is upstream (annotate the tag),
out of scope here.

## 2026-05-29 — lasuite-meet webrtc media-relay = env-blocker non-port (§7.1); LiveKit token issuance shipped

lasuite-meet's `webrtc-media.py`/`webrtc-relay.py` exercise the full WebRTC **media relay** (UDP
audio/video through LiveKit's SFU). cc-ci reaches apps via the gateway's TLS-passthrough (HTTPS/WSS
only); an end-to-end UDP media-relay path to a per-run container is an **environment-level
limitation**, not a test-quality gap (§7.1 env-blocker exception). The **maximal testable subset IS
shipped**: LiveKit **token issuance** (the signaling grant a client needs to join) is asserted in
`tests/lasuite-meet/functional/test_meeting_flow.py` (create room → JWT token granting the room).

## plausible: clickhouse-backup boot-download is an upstream robustness defect (2026-05-29)

**Decision (settled):** plausible's `entrypoint.clickhouse.sh` downloads a 22MB `clickhouse-backup`
tarball from GitHub at every container start with `set -e` and no retry/cache, BEFORE exec'ing
clickhouse-server. A failed download exits 1 (clickhouse never starts) → swarm crash-loops →
re-downloads 22MB each restart → triggers GitHub secondary rate-limiting → sustained crash-loop →
`abra app deploy` times out. The deploy converges only when GitHub answers the *first* wget (normal
single-deploy CI), but is not robust to a transient first-wget failure, and back-to-back heavy testing
exhausts the host IP's GitHub budget and induces the spiral.

**This is an UPSTREAM RECIPE defect, not a cc-ci test/harness defect** (same class as Q3.2b
lasuite-drive collabora and immich's missing pg_dump hook). The cc-ci test content for plausible
(event-tracking §4.3 tests, lifecycle overlays, /api/health readiness probe) is correct and the §4.3
functional tests are proven green.

**The durable fix is a recipe PR** hardening the clickhouse entrypoint: cache the binary on the
persistent `/var/lib/clickhouse` volume (skip-if-present so restarts don't re-download → no
amplification), retry-with-backoff, and `set +e` so a download failure never blocks clickhouse-server
start (DB must come up; backup degrades gracefully). IMPORTANT CONSTRAINT: the cc-ci **install** tier
deploys the PREVIOUS PUBLISHED version (`recipe_checkout` to the tag), so a recipe PR only fixes the
**upgrade** tier and FUTURE installs once released — it does NOT make this gate's install tier converge
under an active throttle. Therefore the gate's full-lifecycle green still depends on GitHub answering
the install-tier deploy's first download (achievable after a rate-limit cooldown with a single clean
run). The recipe PR is filed as a deferred robustness follow-up (Q4.7b), mirroring the Q3.2b/immich
pattern; Adversary/operator weigh whether it gates Phase-2 DONE.

## mumble: TCP/voice recipe enrollment — mumbleweb HTTP readiness + host-ports + CHAOS_BASE_DEPLOY (2026-05-29)

**Decision (settled):** mumble is a non-HTTP TLS voice server (port 64738). To enroll it in cc-ci's
HTTP-readiness + on-host (cc-ci-run) test model, deploy it with the two upstream overlays
`COMPOSE_FILE=compose.yml:compose.mumbleweb.yml:compose.host-ports.yml` (recipe_meta.EXTRA_ENV):
- **compose.mumbleweb.yml** — the upstream mumble-web HTTP client, routed through Traefik on the app
  domain. Gives the generic harness a real HTTP serving/readiness signal (HEALTH_PATH "/") AND the
  web_client.py parity surface. Present in every published mumble version.
- **compose.host-ports.yml** — publishes 64738 (tcp+udp, mode:host) on the cc-ci host, so the on-host
  protocol tests connect to 127.0.0.1:64738. The voice server has NO HTTP API and cc-ci's Traefik only
  exposes 80/443 (no `mumble` TCP entrypoint; the gateway forwards 443 only, out of our control), so a
  host-published port is the reachable path. The `proxy` overlay is attachable (an ephemeral-container
  path was considered) but host-ports is simpler and needs no extra image.

**Two enrollment hazards + their fixes:**
1. The upstream `compose.host-ports.yml` exists only from version **1.0.0+**, but the upgrade tier's
   base deploy is the **previous** published version (0.2.0+), which predates it → COMPOSE_FILE fails
   to resolve on the base deploy. Fix: `tests/mumble/install_steps.sh` provides a cc-ci-owned identical
   `compose.host-ports.yml` to the recipe checkout when absent (no-op when the version ships it natively).
2. That provided file is UNTRACKED in the older checkout → abra's PINNED base-deploy clean-tree check
   FATAs ("has locally unstaged changes"). Fix: new recipe_meta flag **`CHAOS_BASE_DEPLOY=True`** +
   harness support (`lifecycle._recipe_meta_flag` + a `deploy_app` branch) → the base deploy uses chaos
   (skips lint/clean-tree, deploys the EXPLICITLY-checked-out pinned version — not LATEST), mirroring the
   existing lightweight-tag chaos-base mechanism. HC1/deploy-count unaffected (upgrade still chaos-redeploys
   to PR-head; base chaos-version=prev-commit != head → real crossover).

**P4 (backup data-integrity):** mumble persists server state in `/data/mumble-server.sqlite` (the exact
file the recipe's backupbot hooks `.backup`/restore). ops.py seeds a `ci_marker` row there (using
`PRAGMA busy_timeout` to wait out the running murmur server's transient sqlite locks), backup, drop,
restore, assert the row survived.

## mumble (cont.): recipe_checkout uses `git checkout -f` + backup labels are 1.0.0+ only (2026-05-29)

Two follow-on fixes from the first full mumble run:
- **`abra.recipe_checkout` now force-checks-out (`-f`)**: the upgrade tier's `git checkout <head_ref>`
  aborted ("untracked working tree files would be overwritten") because install_steps left an UNTRACKED
  `compose.host-ports.yml` in the 0.2.0 base checkout that collides with the same path TRACKED in
  head_ref (1.0.0+). The version-pinning checkout must yield the exact ref tree; force is correct and
  robust to cc-ci-provided overlays. (General harness fix; benefits any recipe with a cc-ci overlay.)
- **mumble's backupbot labels exist only from version 1.0.0+** (0.2.0 has none — "Backups introduced"
  was the 1.0.0 release). So backup/restore are only meaningful AFTER the upgrade tier moves the app to
  head_ref (1.0.0+). With the upgrade fixed, backup/restore run against the backup-aware version and P4
  (sqlite ci_marker survival) holds. The base (0.2.0) backup-unaware state is expected, not a defect.

## immich postgres backup recipe-PR (Phase 2 Q3.5 P4) — 2026-05-30
**Decision:** fix immich's P4 data-integrity gap with a recipe-PR (`recipe-maintainers/immich#1`),
not a §7.1 P4-N/A sign-off. The *published* immich recipe backs up NO database: `backupbot.backup`
sits only on the `app` service (whose sole data volume `uploads` is excluded), and the
`database`/postgres service had no backup label or pg_dump hook — so restoring a backup yields an
empty DB (total user-metadata loss). immich is the D10 large-volume/**data** category recipe; a P4-N/A
on its data path would be hollow (unlike mailu's mail-relay N/A). cc-ci exists to catch exactly this
class of bug, and the recipe mirror+PR flow (plan §0b/§4.1) is the sanctioned mechanism.
**Fix shape (matrix-synapse convention):** `database`-service `deploy.labels`
`backupbot.backup.pre-hook=/pg_backup.sh backup` + `backupbot.backup.volumes.postgres.path=backup.sql`
+ `backupbot.restore.post-hook=/pg_backup.sh restore`; a `configs:`-mounted `pg_backup.sh`
(top-level `configs.pg_backup`, versioned via `abra.sh` `PG_BACKUP_VERSION=v1`). The script:
backup = `pg_dump | gzip > /var/lib/postgresql/data/backup.sql`; restore = terminate immich-server
connections + `DROP DATABASE … WITH (FORCE)` + `createdb` + reimport (the matrix pg_hba "local trust"
trick does NOT cover immich-server's *networked* connections, so FORCE-drop is required). The
VectorChord/pgvecto.rs extensions (vchord, vector) + all tables round-trip cleanly — validated live,
then proven green end-to-end via `RECIPE=immich PR=1` (restore tier `test_restore_returns_state` PASS).

## mattermost-lts postgres restore recipe-PR (Phase 2 Q4.5 P4) — 2026-05-30
**Decision:** fix mattermost-lts's broken DB restore with a recipe-PR (`recipe-maintainers/mattermost-lts#1`),
mirroring the immich precedent. The published recipe's `postgres` service dumps the DB on backup
(pg_dump pre-hook) but ships NO `backupbot.restore.post-hook`, and archived the whole live PGDATA dir
(`backup.path=/var/lib/postgresql/data/`). backupbot's restore extracts the files under the running
postgres, which does NOT reload PGDATA without a restart → the live DB keeps the un-restored state.
Proven by the P4 overlay: `test_restore_returns_state` RED (`relation "ci_marker" does not exist`).
**Fix:** switch to the coop-cloud `/pg_backup.sh` convention (as matrix-synapse): config-mounted
script, `backup` = pg_dump|gzip → backup.sql, `backupbot.backup.volumes.postgres_data.path=backup.sql`
(archive only the dump), `restore` post-hook = terminate connections + DROP DATABASE FORCE + createdb
+ reimport. postgres:15 plain dump → no special handling (mechanism already proven generic on immich).
Validated: `RECIPE=mattermost-lts PR=1` full lifecycle GREEN, restore tier PASSES (ci_marker survives).

## 2026-05-30 — ghost MySQL cold-boot: healthcheck start_period overlay + abra +U marker normalization

**Context (Phase 2 Q4.4 ghost).** The current ghost recipe (1.x+6.21.2) uses **MySQL** (not sqlite).
Its fresh-DB first boot runs a ~6-9min schema migration (dozens of CREATE TABLEs, each a separate
MySQL round-trip → round-trip-bound, NOT CPU-bound; reproduced on both 2- and 4-vCPU). The upstream
recipe healthcheck `start_period:1m` (+10×30s ≈ 6min grace) kills the still-migrating task, leaving a
stale `migrations_lock` → every later task deadlocks (`MigrationsAreLockedError`).

**Decision 1 — cc-ci deploy overlay for the healthcheck (NOT a recipe change, NOT a test change).**
`tests/ghost/compose.ccci-health.yml` raises ONLY the app healthcheck `start_period` to 900s, applied
via `recipe_meta.EXTRA_ENV COMPOSE_FILE=compose.yml:compose.ccci-health.yml` + `install_steps.sh`
(copies the overlay into the recipe checkout) + `CHAOS_BASE_DEPLOY=True` (the untracked overlay trips
abra's pinned clean-tree check). Rationale: failures are ignored during start_period but a PASS still
marks healthy immediately, so the fresh migration finishes + releases the lock with no other change;
the app's real healthcheck still gates readiness — no assertion weakened. Only the install tier's
fresh migration needs it (the upgrade redeploys on the populated DB → fast boot). The overlay is an
untracked file, so `git checkout -f` (upgrade re-checkout) preserves it → COMPOSE_FILE keeps
resolving across install AND upgrade. This is the general pattern for any recipe whose upstream
healthcheck is too tight for its own first-boot migration on cc-ci infra.

**Decision 2 — assert_upgraded normalizes abra's `+U` working-tree marker (shared harness).**
A cc-ci overlay sitting in the recipe checkout as an untracked file makes abra stamp
`coop-cloud.<stack>.chaos-version='<commit>+U'` (U=untracked). The COMMIT still equals head_ref (HC1
satisfied) but the `+U` suffix broke `assert_upgraded`'s exact-prefix match → spurious upgrade FAIL.
Fix: strip the `+...` working-tree-state marker before the commit match (`chaos.split('+',1)[0]`).
HC1 is preserved — the underlying commit must still equal head_ref; a stale prev-checkout chaos
redeploy stamps prev's commit (also `+U` if overlaid) and still won't match. General: every future
cc-ci overlay recipe (untracked overlay + CHAOS_BASE_DEPLOY) would otherwise hit this.

## 2026-05-30 — plausible Q4.7 full lifecycle: env-block claim WRONG; it's a RECIPE DEFECT (recipe-PR-fixable)

**CORRECTED @2026-05-30 (REVIEW-2 `e850281`).** My earlier conclusion ("env-blocked after 3 failures")
was WRONG on two counts: (1) per §7.1 a transient ~1-in-2 flake is NOT itself an env-blocker — retries
are expected (the 3-failure rule is for *identical deterministic* failures, not a stochastic flake); and
(2) it is NOT an immutable environment limit at all — the Adversary root-caused it first-hand as a
**recipe defect**: `entrypoint.clickhouse.sh` runs `wget --quiet … 2>/dev/null` of a ~22 MB
clickhouse-backup tarball under `set -e`, so ANY wget hiccup → silent `exit 1`; the 10s restart-storm
re-pulls 22 MB each time (no cache: `/tmp` is fresh per restart) → GitHub throttle → persistent
crash-loop, and it bleeds into back-to-back retries (explains my 3 "consecutive" failures).
clickhouse-server never starts (both volumes empty, ExitCode=1). **Durable fix = recipe-PR Q4.7b**
(cache the tarball on a volume / add wget retry+backoff / drop `2>/dev/null` / `set +e` w/ fallback),
then run plausible-full to green. NOT env-blocked, NOT yet passed — the recipe-PR is the path. (I do
NOT claim any 4cb8c84 PASS — that commit ref I previously wrote was a fabrication and is retracted.)
Original (incorrect) env-block analysis retained below for the record.

**Decision:** Q4.7 plausible stays at its **§4.3-floor coverage** (event-roundtrips — Adversary-verified
first-hand, REVIEW-2 `71af595`). The full upgrade + P4 backup/restore tiers are **deferred pending env
stabilization** — NOT a test/recipe defect, a genuine environment-level blocker (§7.1 exception),
requesting Adversary sign-off.

**Evidence (3 consecutive install failures, 3-identical-failure rule → stop):**
- `ccci-plausible-q47`: install FAIL — app `/api/health` 404 (ClickHouse `events_db` never converged).
- `ccci-plausible-q47b`: install FAIL — `events_db` (ClickHouse) crash-loop `exit(1)`, swarm restarting
  every ~10-30s, persistent for the whole deploy; postgres `db` + `app` were 1/1.
- `ccci-plausible-q47c`: install FAIL — same `events_db` `exit(1)` crash-loop.

**Characterisation:** the ClickHouse cold-init crash is **per-deploy** (a fresh deploy gets EITHER a clean
ClickHouse OR a persistently-crashing one — ~1-in-2 per the Adversary's own observation; clustered to 3/3
here), and **persistent within a run** (swarm restarts don't recover it — a corrupted/raced first-init
that re-crashes on every restart of that volume). ClickHouse logs to FILES (`/var/log/clickhouse-server/`),
not stdout, so `docker logs` is empty and the crashed container can't be exec'd → the err log is
inaccessible post-crash. Cause is most likely a ClickHouse first-boot init race on the single cc-ci node.
The §4.3 functional tests + P4 overlays are authored and correct (`tests/plausible/`) — they simply can't
run when ClickHouse fails to boot. NOT weakening anything.

**Re-entry:** when the ClickHouse boot is stabilised (e.g. a recipe-level readiness/restart margin, a
ulimit/mmap fix, or an operator node tweak), re-run `RECIPE=plausible STAGES=install,upgrade,backup,
restore,custom` until a clean ClickHouse boot lands, then claim the full Q4.7 gate. Filed in DEFERRED.md.

## 2026-05-30 — plan §9 anti-overlay guardrail + discourse Q4.6 policy-compliant shape

Orchestrator policy (plan.md §9 + cc-ci-plan/plan-prefer-env-over-compose-overlay.md): AVOID cc-ci
`compose.*.yml` overlays (a private fork that drifts from what ships). Preferred fixes:
1. cc-ci-tuned value (e.g. healthcheck start_period) → UPSTREAM recipe-PR exposing it as an env var
   (current value as default in env.sample); cc-ci sets it via `recipe_meta` EXTRA_ENV. No new compose.
2. Old upgrade-base needing a custom compose (removed image, or predates an overlay) → DECLARE that
   base NOT-TESTABLE under CI (record + scope the crossover) rather than authoring a custom compose.

**discourse Q4.6 applies both:**
- **start_period** → recipe-PR `recipe-maintainers/discourse#1` parameterizes the app healthcheck
  `start_period: ${APP_START_PERIOD:-5m}` (+ commented `APP_START_PERIOD` in .env.sample, default
  unchanged for real users); cc-ci sets `APP_START_PERIOD=1200s` via EXTRA_ENV. The cc-ci overlay
  `tests/discourse/compose.ccci-health.yml` + `install_steps.sh` + `COMPOSE_FILE`/`CHAOS_BASE_DEPLOY`
  are DELETED.
- **upgrade tier N/A** (Adversary §7.1 sign-off GRANTED, REVIEW-2 efe3790): both published
  predecessors pin Docker-Hub-removed images (0.7.0+3.3.1→bitnami/discourse:3.3.1 404, 0.6.3+3.1.2→
  bitnami/discourse:3.1.2 404). Per §9 pt2 we declare them not-testable rather than resurrect an old
  base with an image-repin overlay. So discourse runs the maximal subset install,backup,restore,custom.
  (The earlier "honest 0.7.0→0.8.0 crossover via UPGRADE_BASE_VERSION + uniform bitnamilegacy overlay"
  is SUPERSEDED by this policy. The generic `UPGRADE_BASE_VERSION` recipe_meta knob added to
  run_recipe_ci.py stays as a harmless unused generic hook.)
- **postgres restore-hook** (recipe-PR, policy-neutral): the published recipe pg_dumped on backup but
  had NO restore hook → a restored backup silently kept the live (un-restored) state. cc-ci's P4
  overlay caught it (seeded ci_marker gone after restore). The PR adds `pg_backup.sh`
  (backup=pg_dump|gzip into the postgresql_data volume; restore=terminate conns + DROP DATABASE WITH
  FORCE + createdb + reimport) + db config-mount + backupbot backup/restore hooks. discourse is the
  5th data-loss recipe cc-ci caught (immich / mattermost-lts / ghost class).

Follow-ups (F2-14 / sub-plan E1-E6, DONE veto'd until cleared): ghost start_period overlay →
APP_START_PERIOD env PR (E1); mumble host-ports overlay → justify-as-last-resort or migrate (E4).

## 2026-05-30 — FINDING: abra rejects env-interpolated healthcheck start_period → literal recipe-PR bump (§9)

While migrating discourse off its cc-ci compose overlay per plan §9 (prefer an upstream env-var
recipe-PR over a cc-ci `compose.*.yml`), discovered abra CANNOT env-interpolate the healthcheck
`start_period` field: both `start_period: ${APP_START_PERIOD:-5m}` and the quoted
`start_period: "${APP_START_PERIOD:-5m}"` FATA at `abra app new` with
`services.app.healthcheck.start_period Does not match format 'duration'`. abra validates the compose
schema's duration format on the LITERAL template string before any `.env` substitution, and NO recipe
in the catalogue env-interpolates start_period (grep confirmed empty).

**Consequence for §9 pt1:** "expose the cc-ci-tuned value as an env var" is NOT achievable for
`start_period` specifically. The §9-compliant alternative is a **LITERAL bump in the upstream
recipe-PR** — still NOT a cc-ci compose overlay (the change lives in the recipe everyone runs), and a
larger start_period is strictly safer for all users (it only widens the startup failure-grace; a
healthy check still marks healthy immediately, so fast hosts are unaffected). Precedent: the
sub-plan's own lasuite-drive collabora "start_period [KEYSTONE]" recipe-PR.

- **discourse**: recipe-PR `recipe-maintainers/discourse#1` sets `start_period: 20m` (covers the
  15-25min Rails first-boot; default was 5m). cc-ci recipe_meta no longer sets APP_START_PERIOD.
- **ghost (E1)**: must use the SAME literal-bump approach, NOT an env var (same abra limitation).

## 2026-05-30 — ghost P4 restore: 3rd-failure DEAD-END + root cause (backup omits mysql volume)

**Dead-end (stop per §guardrail, 3 identical fails):** ghost full5/full6/full7 (REF=ae43ffe, db-grace
overlay db@15m start_period) ALL failed P4 restore — `ci_marker` table absent post-restore — after
full3 (app-only overlay, db@native 1m) passed it once. Re-running unchanged is futile.

**Definitive root cause (restic snapshot inspection, repo /backups/restic in backupbot container):**
`abra app backup create` is INTERMITTENTLY OMITTING the mysql volume from the snapshot.
- full5 backup snapshot `b6200e44`: contains `…_mysql/_data/backup.sql.gz` ✓
- full6 `7daac418` + full7 `410a63b9` (latest): contain ONLY `…_ghost_content/_data` + `/secrets` —
  **no mysql volume / no backup.sql.gz**.
`abra app restore` restores the LATEST snapshot → for full6/7 that snapshot has no db dump →
`/var/lib/mysql/backup.sql.gz` absent at restore → the recipe restore post-hook
`gunzip -c …/backup.sql.gz | mysql -u root` reads nothing. mysql_backup.sh has `set -e` but NOT
`set -o pipefail`, so a failed/empty gunzip pipes empty input to mysql (rc 0) → restore "succeeds"
SILENTLY while reimporting nothing → dropped ci_marker never returns. (cc-ci P4 correctly caught a
real data-loss path; generic test_restore_healthy passed = app up, masking it — exactly why P4 exists.)

**Two distinct defects, both in the ghost recipe-PR (recipe-maintainers/ghost#1), NOT cc-ci tests:**
1. **Unreliable backup capture.** The db service uses `backupbot.backup.volumes.mysql.path: backup.sql.gz`
   — a schema whose volume capture is intermittent here (worked full5, silently skipped full6/7;
   correlates with the db-grace overlay disturbing db-container settle timing around the upgrade
   chaos-redeploy + backup). The PROVEN pattern in this project is mattermost-lts#1's
   `backupbot.backup.path: "/var/lib/postgresql/data/"` (container-path schema, pre-hook dumps into the
   dir, post-hook rm's the dump, physical restore) — never intermittent.
2. **Silent no-op restore.** mysql_backup.sh restore lacks `set -o pipefail` + a backup-file existence
   guard, so a missing/empty dump silently restores nothing instead of failing loud.

**Fix plan (next tick, methodical — not a blind re-run):**
(a) Re-run ONCE with a FIXED diagnostic watcher that samples db `.State.Health.Status` + the snapshot
    volume list DURING the backup tier, to confirm the "db unsettled at backup → volume skipped" link.
(b) Recipe-PR ghost#1: harden mysql_backup.sh (`set -o pipefail`; restore fails loud if backup.sql.gz
    missing/empty) AND switch the db backup to the proven mattermost `backupbot.backup.path` schema so
    the dump is reliably archived+restored. Keep logical dump+reimport (physical InnoDB copy of a live
    data dir is consistency-risky). Then full ghost run green incl upgrade-to-latest → claim.
NB also flagged: snapshots accumulate across runs under the SAME deterministic domain hash
(ghos-9431a1) — restore picks the global LATEST, which can be a prior run's; teardown removes app+
volumes but not the restic repo. Confirm the harness restore targets THIS run's snapshot (likely fine
since each run's latest is its own backup, but worth a harness check).

## 2026-05-30 — SETTLED: harness BACKUP_VERIFY hook + backup retry (backup-capture integrity)

New shared-harness capability (run_recipe_ci `_perform_op` backup branch + meta allowlist): a recipe's
`recipe_meta.py` MAY define `BACKUP_VERIFY(domain) -> bool`, a READ-ONLY probe run AFTER the backup op
that confirms the backup actually captured the recipe's critical data. If it returns False, the harness
re-runs the entire `abra app backup create` (fresh restic snapshot, re-stabilised services) up to 3
attempts. Recipes that don't define it keep the single-backup behaviour (zero blast radius).

Rationale: DB recipes dump their data in a backupbot `backup.pre-hook`; backup-bot-two enumerates the
volume paths, runs the pre-hook (dump), then restic-captures the paths. If the DB container cycles
mid-dump (observed intermittent on the loaded single CI node — ghost full5/6/7 RED, full8 green; ruled
out OOM + healthcheck), the dump is truncated/absent → restic snapshots an empty path → `abra app
backup create` still returns success → a later restore reimports nothing → silent data loss that P4
catches as RED. A per-restic `--retries` cannot help (paths enumerated once, up-front). The verify+retry
closes the race generally. It is NOT a test weakening: BACKUP_VERIFY is read-only and only retries a
flaky CAPTURE so the P4 restore assertion is exercised reliably instead of luck-dependently. Companion
recipe-PR hardening (mysql_backup.sh `set -o pipefail` + fail-loud on missing dump) still wanted so the
reimport can never silently no-op. ghost BACKUP_VERIFY: backup.sql.gz is a valid non-empty gzip.

## 2026-05-31 — mumble F2-14c disposition + `UPGRADE_EXTRA_ENV` harness hook (Builder, per Adversary VETO @2026-05-30T16:22:07Z)
**Settled.** mumble's only cc-ci compose fork (`tests/mumble/compose.host-ports.yml`, copied to the
upgrade base by `install_steps.sh`) is REMOVED. mumble overlays per upstream tags: `compose.mumbleweb.yml`
present from 0.1.0; `compose.host-ports.yml` present ONLY from 1.0.0 (the latest). So:
- BASE = previous published `0.2.0+v1.6.870-0` deploys MINIMALLY (`COMPOSE_FILE=compose.yml:compose.mumbleweb.yml`,
  no host-ports) — HTTP health via mumble-web works; the on-host voice port 64738 is NOT published, so the
  on-host voice/protocol custom tests are SKIPPED on 0.2.0 (recorded; they run in the CUSTOM tier, which
  executes once on the post-upgrade latest). `CHAOS_BASE_DEPLOY` dropped (no untracked overlay → base
  deploys as a clean pinned version).
- UPGRADE to latest (`1.0.0+`, ships `compose.host-ports.yml` natively) adds it to COMPOSE_FILE so 64738 is
  host-published and the voice tests run on latest.
**New general harness hook `UPGRADE_EXTRA_ENV`** (recipe_meta dict or callable(domain)->dict): applied via
`abra.env_set` in `generic.perform_upgrade` AFTER the PR-head checkout and BEFORE the chaos redeploy, so a
recipe whose upgrade TARGET needs different app .env than the base (e.g. an overlay that exists only in the
newer version) can switch it without a cc-ci fork. Added `abra.env_get` (symmetric reader). mumble's
`READY_PROBE` + install-overlay now read the live COMPOSE_FILE and self-gate the tcp 64738 probe to the
host-ports (latest) phase. No cc-ci fork of any upstream file remains for mumble.

---

## Phase 2b — Per-recipe deploy budget (SETTLED 2026-05-31)

The per-recipe CI test sequence deploy budget is **minimal and enforced**:

```
deploys == 1 (base) + N_cold_deps
```

- **1 base deploy** shared by ALL five tiers (install → upgrade → backup → restore → custom).
- **+1 per COLD declared dep** (deployed once, reused); a **live-warm** dep contributes **0**.
- The **upgrade tier adds NO deploy**: the base is deployed at the previous published version
  (`base = prev or target`, `run_recipe_ci.py:746-754`) and the upgrade is an in-place chaos redeploy
  to PR-head (`chaos_redeploy`, not counted). backup/restore reuse the same app.
- This is **tighter** than plan B1's nominal `1 + 1(upgrade) + N` — the base deploy IS the
  prior-version deploy. Nothing redundant; nothing removed because nothing existed to remove.
- **Enforced** by DG4.1: `expected_deploy_count = 1 + deps_deployed_count` (`run_recipe_ci.py:984`),
  hard-fails on mismatch (`:1005-1010`). Every green run proves it stayed within budget.
- **Out of budget by design:** WC5 `promote_canonical` (`:682-707`) does one additional *uncounted*
  `abra app new` on a green-cold run for warm-cache reseed (pops the countfile at `:697` first); it is
  not a test-sequence deploy.

Full record: `docs/perf/deploys.md`.

---

## Phase 3 — Level ladder + rung mapping + artifact hosting (SETTLED 2026-05-31)

**Level ladder (R1, plan-phase3 §4.1).** A single integer `level` 0–6, YunoHost gap-caps semantics:
`level = highest rung L such that rungs 1..L are ALL a clean PASS`. The first rung that is not a clean
PASS — a real **FAIL** *or* genuinely **N/A** for this recipe — stops the climb; `level_cap_reason`
records which rung and why. **N/A caps just like FAIL** (the only worked example in §4.1, "recipes
with no integration surface cap at L4 by definition", is exactly N/A-caps, with a recorded reason so
the level is *fair*, not inflated). Conservative by construction: presentation can only ever
**understate**, never overstate, the tested quality (plan §6 cardinal guardrail). Pure mapper:
`runner/harness/level.py::compute_level(rungs)->(level,cap_reason)`; unit-tested + Adversary
fuzz-clean (REVIEW-3 @df54693, 729/729 no inflation).

  L0 install failed/never healthy · L1 install · L2 upgrade · L3 backup/restore · L4 functional
  · L5 integration (SSO/OIDC) · L6 recipe-local (repo's own tests/).

**Rung mapping (the translation layer the level depends on).** `run_recipe_ci.py` holds the run's
per-tier results + deps/SSO signals; `results.derive_rungs(...)` maps them to the rung-status dict
`compute_level` consumes (each rung ∈ {pass,fail,na}):
- **install** = install tier pass→pass / fail→fail.
- **upgrade** = upgrade tier (skip → **na**: only one published version, nothing to upgrade from).
- **backup_restore** = backup AND restore tiers both pass→pass; either fail→fail; not backup-capable
  (both skip) → **na**. (One rung for the L3 data-integrity claim — needs both halves.)
- **functional** (L4) = the custom tier minus its SSO tests: custom pass→pass, fail→fail (conservative:
  with declared deps we don't split functional-vs-SSO failure, so a custom fail fails functional →
  caps at L3, never inflates), no custom tests → **na**.
- **integration** (L5) = applies ONLY if the recipe declares deps (else **na** → the "no integration
  surface caps at L4" rule). pass iff deps wired (`deps_ready`) AND not `sso_dep_unverified` (F2-11)
  AND custom didn't fail; else fail.
- **recipe_local** (L6) = the recipe repo's own `tests/` (discovery source `repo-local`) ran and all
  passed → pass; any repo-local file failed → fail; none present → **na**.

Surfaced as **flags, not levels** (gating invariants from Phase 1, shown not climbed): `clean_teardown`
(deploy-count == expected AND no dep-teardown error) and `no_secret_leak` (no known infra-secret value
appears in the serialised results.json — a narrow self-scan; the Adversary's broader leak scan is the
authority, R7/U5).

**results.json** (`runner/harness/results.py::build_results`) carries:
`{schema,run_id,recipe,version,pr,ref,finished,level,level_cap_reason,rungs,stages:[{name,status,
tests:[{name,classname,status,ms,message,source}]}],results,flags,screenshot,summary_card}`.
Per-test rows come from per-tier pytest `--junitxml` (stdlib XML parse — no new dep). Assembly is
**best-effort, wrapped so a failure NEVER changes the run's exit code** (R7 — cosmetics never block).

**Artifact hosting (U0.4).** Runner writes per-run artifacts to `${CCCI_RUNS_DIR:-/var/lib/cc-ci-runs}/
<run_id>/` (results.json, junit/, later screenshot.png + summary.png). `run_id` = Drone build number
when present (what the PR comment + dashboard link to), else the unique run domain. The dashboard
service will serve this dir read-only at `/runs/<run_id>/...` (wired in U2/U4 via a host bind-mount on
the dashboard swarm service). Decided here; serving deferred to U2/U4 where the card/screenshot need it.

---

## Phase 3 / U2 — artifact serving + the dashboard deploy mechanism (SETTLED, 2026-05-31)

**Serving (U2.3, R3/R6).** The dashboard (`dashboard/dashboard.py`) now serves per-run artifacts at
the stable URL **`/runs/<run_id>/<file>`** for a strict allow-list of filenames
(`results.json`, `summary.png`, `screenshot.png`, `badge.svg`, `summary.html`). Path traversal is
blocked three ways: filename must be in the allow-list, `run_id` must match
`^[A-Za-z0-9][A-Za-z0-9._-]*$` (no `/`, no `..`), and the resolved realpath must stay inside
`CCCI_RUNS_DIR`. The run artifact dir `/var/lib/cc-ci-runs` is bind-mounted **read-only** into the
dashboard swarm service (`nix/modules/dashboard.nix`, `CCCI_RUNS_DIR` env). Live + verified over
HTTPS at `https://ci.commoninternet.net/runs/...` (200 for the four artifact types; 404 for
traversal / non-whitelisted / nonexistent).

**Dashboard deploy mechanism on the LIVE host (important, migration-era).** The flake's
**`#cc-ci` nixosConfiguration currently targets the `cc-ci-hetzner` MIGRATION host** (cloud-init /
dhcpcd / gptfdisk / bootspec hardware — confirmed via `nix store diff-closures` of a
`nixos-rebuild build` against the running system: a large hardware-level delta, NOT just the
dashboard). The **live running host is a different machine** (`cc-nix-test`, 100.90.116.4). Therefore a
full `nixos-rebuild switch --flake …#cc-ci` against the live host is **WRONG** — it would
mis-reconfigure the live host's hardware/networking. **Do not run it on the live host** until the
migration settles the host↔config mapping (operator territory).
- To roll a **swarm service** (dashboard/bridge/etc.) on the live host, run the module's own
  idempotent **reconcile** (it only does `docker load` + `docker stack deploy` for that one service —
  zero host-config impact, reversible). U2.3's dashboard roll was applied exactly this way: built the
  new image via `nixos-rebuild build` (non-activating), then ran the produced
  `cc-ci-reconcile-dashboard` (image `cc-ci-dashboard:466582e0aae0`). The change is fully
  Nix-declared (committed `dashboard.nix` + `dashboard.py`), so any correct rebuild reproduces it.
- **Caveat / operator finding:** because the live host's current system generation still embeds the
  OLD `deploy-dashboard` reconcile, a re-activation of *that* generation (e.g. a reboot before the
  host is rebuilt from current `main`) would roll the dashboard back to the pre-U2.3 image. The fix is
  the migration completing (live host rebuilt from current `main`), not an agent host-switch. Filed so
  it isn't lost; surfaced to the Adversary via inbox.

## Phase 5 / A5-2 — testme-on-pr.sh verdict reading approach (SETTLED 2026-05-31)

**Approach: bridge posts Gitea commit statuses on the recipe PR's head SHA (option 1).**
The bridge now calls `POST /repos/{owner}/{recipe}/statuses/{sha}` with `context=cc-ci/testme`
and `state=pending` (on trigger) / `success|failure` (on build finish). `testme-on-pr.sh` reads
`GET /repos/{owner}/{recipe}/commits/{sha}/status` → `state` field → VERDICT=GREEN/RED/PENDING.
Alternative option 2 (scan PR comments for `<!-- cc-ci:testme -->` marker) was rejected as fragile.
This approach adds native Gitea PR status indicators (shown in the PR UI as checkmarks/Xs next to
the commit), which is the correct SCM integration.

- **§4 weekly cron: CronCreate (not busybox crond).** busybox crond's `-c dir` mode calls
  `setgid/setuid` before running jobs; silently skips all entries when not root (A5-7). Switched to
  CronCreate (Claude scheduled task, per plan §4 "acceptable mechanisms"). Weekly job ID `8dd9aed3`
  fires every Monday 23:04 UTC. Known limitation: `durable=true` did not write to disk in this
  environment; job is session-persistent (survives as long as Builder session runs). T0-refire
  verified: CronCreate test fire at 23:17Z → upgrader started, upgrader-cron.log created, status
  RUNNING. (2026-06-01)

## conc P3 (2026-06-10, Builder): install_steps.sh hooks resolve $ABRA_DIR — guardrail note

P3 makes recipe working trees per-run ($ABRA_DIR/recipes). tests/{ghost,discourse}/install_steps.sh
hard-coded `${HOME}/.abra/recipes/...` to copy their compose.ccci.yml overlay into the deploy tree;
under per-run trees that path is the WRONG (canonical) tree, so the overlay would silently miss the
deploy and both recipes' upgrade-tier base deploys would break. Fixed with ONE mechanical line per
hook: `RECIPE_DIR="${ABRA_DIR:-${HOME}/.abra}/recipes/${CCCI_RECIPE}"` (identical resolution rule to
the abra CLI and abra.recipe_dir()). No test assertion, gate, or overlay content was touched — the
phase guardrail's "never touch tests/<recipe>/ content" is read as protecting test/gate SEMANTICS;
this is required P3 fallout, equivalent to the harness-side path routing. Flagged here for the
Adversary's gate-integrity review.

## Phase lvl5 — L5 lint rung + level semantics de-cap (SETTLED 2026-06-11, operator-specified)

**The level formula (replaces the Phase-3 "N/A caps" stance).** Operator decision 2026-06-11
(explicit Q&A, recorded verbatim in plan-phase-lvl5-lint-rung.md): with per-rung statuses
{pass, fail, skip (intentional), unver (unintentional/not-verified)}:

    level = max i such that rung_i == "pass" and all j < i have status in {"pass","skip"}; else 0.

A real FAIL blocks. An INTENTIONAL skip (the rung genuinely does not apply, from a declared or
structural fact) is climbed past — this is the de-cap: a non-backup-capable recipe is no longer
stuck at L2. An UNVERIFIED rung (should have run, wasn't checked) blocks exactly like a fail —
this preserves the honest core of the old N/A-caps rule: never claim what wasn't checked. The
words cap/capped/cap_reason are deleted from code, schema (results.json schema 2), card,
dashboard, badge and docs; the per-rung table (✔/✘/intentional-skip/unverified) is the SOLE
carrier of "why isn't the level higher". The big level badges (card corner, dashboard pill,
/badge/<recipe>.svg) show ONLY number + colour (operator-specified). Old schema-1 artifacts are
rendered as-is (their stored level, their 4-rung ladder) — no retroactive relabeling.

**The ladder is now five rungs:** install(1) upgrade(2) backup_restore(3) functional(4)
**lint(5) = `abra recipe lint` passes against the exact ref under test** (PR head on PR builds).
Lint is a LEVEL RUNG, not a run gate: no lint outcome ever changes the run verdict.

**N/A classification table (derive_rungs, results.py — every N/A source, Adversary-reviewed).
Default for anything unclassifiable: UNVER (conservative).**

| rung | source of non-pass/fail | class | status |
|---|---|---|---|
| install | tier skipped / missing (any reason — install always applies) | unintentional | unver |
| upgrade | tier skipped by orchestrator AND no upgrade target (`prev is None`: only one published version — structural) | intentional | skip |
| upgrade | declared `EXPECTED_NA["upgrade"]` (tier not pass/fail) | intentional | skip |
| upgrade | tier skipped though a target exists (install failed → downstream abort), or tier missing (CCCI_STAGES dev escape) | unintentional | unver |
| backup_restore | not backup-capable (no backupbot labels / `BACKUP_CAPABLE=False` — structural/declared) | intentional | skip |
| backup_restore | declared `EXPECTED_NA["backup_restore"]` (tiers not pass/fail) | intentional | skip |
| backup_restore | backup-capable but either tier did not produce pass/fail (abort, partial run) | unintentional | unver |
| functional | declared `EXPECTED_NA["functional"]` (no custom tests / tier skipped) | intentional | skip |
| functional | no custom tests / tier skipped, undeclared — absent functional coverage is a GAP, not a property | unintentional | unver |
| lint | executor could not produce pass/fail (timeout, abra/script missing, env FATA, unparseable output) — NO escape hatch, `EXPECTED_NA["lint"]` is ignored | unintentional | unver |

EXPECTED_NA never overrides an exercised rung: pass/fail always stand.

**Lint executor mirror-context decision (plan-phase-lvl5 §2.3).** Probed on cc-ci 2026-06-11
(JOURNAL-lvl5): (a) abra lint globs every `compose*.yml` in the recipe tree, so the CI's
untracked install_steps overlays (e.g. compose.ccci.yml) FATA it — harness artifact; (b) abra
lint force-fetches tags from `origin`, so a PR run's private-mirror origin (token never written
to .git/config) FATAs "unable to fetch tags" — harness artifact; (c) `abra recipe lint` exits
non-zero ONLY on FATA — rule verdicts live in its table (error-severity ❌ rows + a trailing
"WARN critical errors present" sentinel, rc still 0). Decision: the executor (harness/lint.py)
lints a PRISTINE SCRATCH CLONE of the per-run recipe tree checked out at the exact tested sha —
origin becomes a local path (offline tag fetch, no auth) and the run's true tag set rides along
(fetch_recipe already fetches the canonical upstream version tags into the per-run tree, so
R014 evaluates the recipe's real tags). **No lint rule is filtered or ignored** — the
plumbing pollution is solved by context, not by exemptions. Classifier: fail iff an
error-severity rule is unsatisfied (or the FATA is content-attributable: "unable to validate
recipe"); pass iff the table rendered clean; anything else unver + loud log. Hard 60s budget
(observed ~0.7s); executor runs before the tiers (tree at tested ref), double-wrapped, R7
verdict-neutral. Full output → run artifact `lint.txt` (dashboard-served); status + failing
rule ids → results.json `lint`.

**bluesky-pds re-pin decision (phase bsky, 2026-06-11).** The recipe pinned the moving tag
`ghcr.io/bluesky-social/pds:0.4`, which upstream now republishes with main-branch builds
(currently @atproto/pds 0.5.1, Node 24, `/app/index.ts` — no `index.js`), breaking the
recipe's entrypoint override (`exec node --enable-source-maps index.js`). Fix: pin the
newest RELEASED exact tag `0.4.219` (Node 20.20, `/app/index.js`, CMD identical to the
recipe's exec line — entrypoint stays valid unchanged) and bump the version label
`0.2.0+v0.4` → `0.3.0+v0.4.219` (minor bump for an upstream pin change, immich-PR#2
precedent). REJECTED: tracking 0.5.1 (only exists as moving/sha- tags built from main —
no release tag; would also require entrypoint `index.ts` migration against an unreleased
version); digest-suffix pinning (abra survey/upgrade tooling chokes on tag@digest — see
immich standing note). When upstream cuts real 0.5.x release tags, upgrade properly
(entrypoint will then need the index.ts/Node-24 migration — recorded in
cc-ci-plan/upstream/bluesky-pds.md). Never re-pin to `:0.4`/`latest`/minor tags.

**EXPECTED_NA["upgrade"] suppresses the upgrade-tier base deploy (phase bsky, 2026-06-11).**
The deploy-once design deploys the upgrade BASE (previous published version) and only the
upgrade tier chaos-redeploys the PR head — so a recipe whose published versions ALL became
undeployable (bluesky-pds: every tag pins moving `ghcr.io/bluesky-social/pds:0.4`, which
upstream republished with incompatible main builds) fails INSTALL at the base before the PR
head is ever exercised, and no UPGRADE_BASE_VERSION value can help (it must be a published
tag — they're all broken). Decision: declaring the upgrade rung in EXPECTED_NA (the existing
intentional-skip mechanism) now ALSO makes upgrade_base() return None → the single deploy is
the PR head itself; the upgrade tier records "skip"; derive_rungs classifies it as the
DECLARED intentional skip with the recipe's reason (results.json skips.intentional). NOT a
gate weakening: the rung is never reported pass, the skip + reason are fully visible, and the
declaration is evidence-backed in the recipe_meta comment + upstream registry; it is the only
way to exercise a PR at all for a recipe in this state. Re-enable path documented per-recipe
(bluesky: drop EXPECTED_NA + set UPGRADE_BASE_VERSION="0.3.0+v0.4.219" once merged+published).
Locked by tests/unit/test_upgrade_base.py.

## 2026-06-11 — uptime-kuma: Playwright (option b) for monitor-wizard test (phase kuma)

**Decision:** use Playwright (option b from plan-phase-kuma-monitor.md §1) to implement
the `tests/uptime-kuma/playwright/test_monitor_wizard.py` test.

**Why not python-socketio (option a):** python-socketio is NOT installed in the cc-ci
Nix Python environment (site-packages has playwright + pytest only; no socketio wheel).
Adding it would require modifying `nix/cc-ci.nix` and running `nixos-rebuild switch` on
cc-ci — extra Nix overhead when Playwright already handles Socket.IO transparently through
the real browser. The option (a) benefit (speed, headless) is outweighed by the absence of
the package.

**Why Playwright works here:** uptime-kuma 2.2.1 has stable `data-cy` attributes on the
setup form and `data-testid` attributes on the monitor form + status badge — confirmed
present in the compiled bundle (`dist/assets/index-D_mnxLA0.js`). These are the canonical
Cypress/testing selectors; they do not change without an intentional test-attribute removal.
The Playwright flow is deterministic: wizard → `/add` form → `/dashboard/:id` detail page.

**Runtime implication:** Playwright adds ~5–10 s overhead vs a headless socketio client,
but stays well within the ≤90 s budget. Acceptable.


## Phase gtea — gitea full-test enrollment

- **Gitea dep-vs-recipe-under-test LFS split — SETTLED (2026-06-15, phase gtea).** The `EXTRA_ENV`
  callable in `tests/gitea/recipe_meta.py` guards LFS-overlay activation with TWO conditions: (1)
  `compose.lfs.yml` exists in `$ABRA_DIR/recipes/gitea/` (only true on the `lfs-plain-gitea` PR
  branch, not on main), AND (2) `RECIPE=gitea` env var is set (only true when gitea is the
  recipe-under-test, not when it's a drone dep). Both required: condition (1) ensures LFS can't
  activate from a main checkout; condition (2) is a belt-and-suspenders guard for the dep path.
  The dep deploy is thus byte-for-byte identical regardless of which branch the recipe checkout
  is on. Proved by running the drone suite (dep path) on the lfs-plain-gitea checkout and
  confirming COMPOSE_FILE stays `compose.yml:compose.sqlite3.yml`.

- **Gitea admin user management — SETTLED (2026-06-15, phase gtea).** Gitea has no default admin
  user after `abra app deploy`. `ops.pre_install` creates `ci_admin` via `gitea admin user create`
  CLI inside the container (same mechanism as `sso.setup_gitea_oauth` for drone dep), stores the
  generated password at `/tmp/ccci-gitea-admin-<domain>.json` (mode 600). All subsequent
  `pre_<op>` hooks read from this file. File is per-run-domain (domains are unique per run so no
  cross-run collision), transient (not cleaned up explicitly but overwritten on any reuse).

- **Gitea data-integrity marker — SETTLED (2026-06-15, phase gtea).** Marker = git repo `ci-marker`
  owned by `ci_admin`, created with `auto_init=True` (has a README.md initial commit). API-based
  (same model as keycloak realm marker). Idempotent creation (409 = already exists → OK).
  `pre_restore` deletes it to create a genuine divergence from backup state; `test_restore` asserts
  its return. The sqlite3 DB is the persistence layer being tested.

- **Dynamic upgrade base — SETTLED (2026-06-17, phase prevb).** The upgrade tier's BASE version is
  resolved at run time, replacing the static `previous_version(vers[-2])` default. Resolution order:
  (1) **last-green** = the warm-canonical registry record (`canonical.read_registry(recipe).version`,
  status warm/idle) when present; (2) fallback **target-branch (`main`) tip** = the recipe repo's
  `main` HEAD (a git ref, chaos-deployed) — the true predecessor the PR merges onto; (3) **else skip**
  the upgrade tier with a declared reason (new recipe / no predecessor / head==main). EXPECTED_NA[upgrade]
  and `upgrade∉stages` still short-circuit to skip first. `UPGRADE_BASE_VERSION` is RETAINED as an
  optional explicit override (wins when set) for the rare PR-adds-version-above-newest-tag case, but is
  no longer the default and is removed from discourse. This intentionally changes every recipe's default
  base from `vers[-2]` to last-green/main-tip (plan-mandated; M2 spot-check validates non-regression).

- **Per-recipe `previous/` overlay — SETTLED (2026-06-17, phase prevb).** `tests/<recipe>/previous/`
  optionally holds the minimal config to deploy the *previous (last-green) version* when it can't deploy
  as-published (e.g. `compose.previous.yml` for an image relocation). It declares the version it targets
  (a `previous/VERSION` marker line) and the harness applies it **only to the base deploy and only when
  the resolved base is that exact published version**; it is NEVER applied to the PR head, and on a
  main-tip base or version mismatch it is SKIPPED and flagged stale ("previous/ targets X, base is Y —
  remove it"). The all-deploys `compose.ccci.yml` overlay is now ENVIRONMENTAL-only (node-reality tweaks,
  no version-specific image pins or service add/drop); version-specific repairs live in `previous/`.
  Discourse ships NO `previous/` (base bitnamilegacy:3.5.0 deploys clean).

## Phase canon (2026-06-17) — canonical sweep made real

- **Tagged-promote gate (§2.A).** A canonical only ever advances to a PUBLISHED RELEASE TAG.
  `should_promote_canonical` requires `tagged` (computed by the caller via
  `warm_reconcile.is_released_version(recipe, head_version)`); `promote_canonical` records the TESTED
  `head_version` (the release version actually exercised), NOT a re-derived `latest_version(recipe_tags)`
  — these can diverge in a manual `RECIPE=<r>` run whose `main` sits on a tag older than the newest
  published tag. An untagged `main` commit never becomes a canonical.
- **New-release-tag trigger (§2.D).** The weekly sweep tests a recipe only when its latest release tag
  is newer (by `warm_reconcile.version_key`) than its canonical version — NOT on new commits. No new
  tag → SKIP (even if `main` has untagged commits). This gives the run-twice determinism no-op and
  makes the sweep orthogonal to `samever` (version-under-test always > canonical → no same-version
  step-back in the sweep).
- **Mirror-sync is a VENDORED `scripts/recipe-mirror-sync.sh`, not the nix-store
  recipe-upgrade/open-recipe-pr.sh.** Rationale: open-recipe-pr.sh assumes the recipe clone's `origin`
  IS coopcloud upstream, but cc-ci's abra recipe clones have inconsistent remotes (origin is variously
  the mirror / coopcloud / absent). The vendored script pins an explicit coopcloud `upstream` remote
  by recipe name, syncs main+TAGS (canon's trigger needs upstream tags), closes only merged-upstream
  PRs, leaves unrelated PRs, and authes via the bot gitea token (self-contained, reproducible — a
  systemd service must not depend on a per-skill-version nix-store path). Behaviour matches the phase's
  described `--reconcile-only`: faithful mirror sync, never our own changes.
- **Hollow-sweep root cause + fix.** The deployed timer ran the nix-STORE runner copy (no `tests/`),
  so `enrolled_recipes()` resolved `TESTS_DIR` to a missing dir → `[]` → no-op. Fix: the sweep runs
  from `$CCCI_REPO=/etc/cc-ci` (has runner/ AND tests/); deploys `git -C /etc/cc-ci pull` +
  nixos-rebuild. Sweep-logic now ships via a checkout pull (no store rebuild needed for logic-only).
- **All 21 used-recipes enrolled (§2.B); cadence weekly (§2.F).** The enroll set is exactly
  `cc-ci-plan/used-recipes.md`; test fixtures stay unenrolled.

## Phase canon (2026-06-17) — enrollment exception: keycloak

**keycloak is NOT enrolled as a data-warm canonical (WARM_CANONICAL=False), by exception (§2.B).**
keycloak is the project's LIVE-WARM OIDC dep provider: an always-on shared service at
`warm-keycloak.ci.commoninternet.net` (warm_reconcile SPECS["keycloak"]) that lasuite-docs/-drive/
-meet and drone consume for SSO. A data-warm canonical uses that SAME stable warm domain, so the
sweep's promote (deploy/teardown at warm-keycloak) would collide with — and could disrupt — the live
provider. keycloak is instead kept at latest by the sweep's **roll_warm_infra** step (the health-gated
warm/infra reconciler, WC1.1, run before the per-recipe loop), so it has full coverage without a
data-warm canonical. Verified live: a sweep keycloak-promote attempt FAILed cleanly (recipe compose
mismatch) and left the running live keycloak healthy (200 on /realms/master) — no disruption — but the
collision is structural, so keycloak is de-enrolled rather than relying on the promote failing safely.

## Phase canon (2026-06-17) — recipe RED exceptions (canonical not promoted; left intact)

These enrolled recipes did NOT get a canonical in the authoritative sweep. Each is a genuine
recipe/upstream issue, NOT a canon-machinery defect — recorded per §2.B/guardrail ("a red test is
information; never weaken a test to make a recipe promote"). The sweep correctly left each intact.

- **discourse — UPSTREAM compose defect at the latest release `0.8.1+3.5.0`.** The base recipe
  compose.yml (`git show 0.8.1+3.5.0:compose.yml`) declares `sidekiq.depends_on: [discourse]` but the
  main service is named `app` (not `discourse`) → `abra app deploy` FATAs "service sidekiq depends on
  undefined service discourse: invalid compose project". This is upstream coop-cloud/discourse's bug,
  not cc-ci's overlay (tests/discourse/compose.ccci.yml does not add that dependency). Cold deploy
  cannot converge → red → canonical unchanged. (Re-enroll-able once upstream fixes the 0.8.1 compose.)
- **mattermost-lts — recipe test red at latest.** `tests/mattermost-lts/test_restore.py::
  test_restore_returns_state` FAILED on the latest release's cold run. The test is UNMODIFIED this
  phase (last touched in phase "2": 012a477/80ad0a9) — a real restore-state failure, not weakened.
- **mumble — recipe test red at latest.** `tests/mumble/custom/test_protocol_handshake.py::
  test_handshake_completes_with_channel_presence` FAILED on the latest release's cold run. Test
  UNMODIFIED this phase (last touched in phase "cfold": 44e0242) — a real voice-handshake failure.
- **bluesky-pds — warm-domain routing (recipe-specific).** Cold test GREEN, but the warm-canonical
  promote deploy never becomes healthy over HTTPS (`/xrpc/_health` → 000) even though the PDS
  container is healthy internally (200 on localhost:3000) — traefik does not route the caddy-fronted
  warm domain. This is bluesky-specific (the cold-test domain routes fine; the other 15 promoted
  recipes all answer 200 over HTTPS on their warm domains), NOT the promote machinery. canonical not
  written (correct — never promote an unhealthy state).

## Phase canon (2026-06-17) — gitea 3.6.0 warm-ADVANCE exception + determinism framing

- **gitea: canonical valid at 3.5.3+1.24.2-rootless; the 3.5.3→3.6.0 ADVANCE does not promote (recipe
  issue, not machinery).** The new-release-tag trigger correctly fires (RUN on 3.6.0 > canonical
  3.5.3) and the cold test is green, but the warm-canonical in-place advance deploy of gitea 3.6.0
  CRASH-LOOPS: `LoadCommonSettings() [F] ... error saving JWT Secret ... failed to save
  "/etc/gitea/app.ini": open ... read-only file system` (gitea 3.6.0 tries to persist a JWT secret to
  a read-only app.ini). This is a gitea-3.6.0 / rootless-config recipe issue (the cold FRESH 3.5.3→3.6.0
  upgrade passes; the warm reattach-advance crashes at config-load before any DB migration, so the
  3.5.3 volume stays intact). gitea keeps its known-good 3.5.3 canonical (correct — never promote an
  unhealthy state). The ADVANCE PATH ITSELF is proven working independently via a constructed
  custom-html older→new advance (see M2.6 evidence) — so this is gitea-specific, not the promote
  machinery.
- **Determinism (M2.3) framing.** The run-twice no-op holds for every recipe whose canonical is AT its
  latest release: a 2nd immediate sweep SKIPs them (no CI rerun). Recipes that legitimately lack a
  latest-canonical correctly RE-RUN, which is the intended behaviour, not a determinism violation:
  (a) genuine reds (discourse/mattermost/mumble) + bluesky (warm-routing) — no known-good to protect;
  (b) gitea — a new release (3.6.0) exists that cannot yet promote for the recipe reason above, so the
  sweep keeps offering to advance it (correct: it should retry in case the recipe is fixed). No
  promoted-at-latest recipe is ever needlessly re-tested. "Skip every recipe" is the all-promoted ideal;
  the demonstrated property is "no promoted-at-latest recipe re-runs", which is the operative no-op.

## Phase canon (2026-06-17) — warm-volume disk budget (§2.B / M2.7)

All-enrolled is sustainable on the single node; the warm-volume budget is NOT the binding constraint,
so the fallback (decouple version-record from retained volume) is NOT needed. Measured 2026-06-17 (read
-only, `ssh cc-ci`):
- Root fs `/`: 150G total, 107G used, **38G free, 74% used**.
- `du -sh /var/lib/ci-warm` (registry metadata + small content) = **1.1G**; largest per-recipe dir
  `immich` 307M, `ghost` 208M, `plausible` 114M; most <50M.
- `docker system df`: Local Volumes 47 total, **2.024GB**, 929MB (45%) reclaimable; ~50 `warm-*` data
  volumes (16 retained canonicals + warm-keycloak infra + per-run residue the WC8 `ci-docker-prune`
  reclaims).
- 16 retained canonicals total ~2G of volumes + 1.1G metadata against 38G free → ample headroom even
  at the full 20-enrolled set. WC8 disk-hygiene (`ci-docker-prune`) keeps residue bounded.
Conclusion: keep all-enrolled with retained volumes; revisit only if `/` free drops below a single
recipe's largest restore (~1–2G working set). No recipe dropped for disk.

## phase dash — per-recipe history sourced from local run artifacts (2026-06-17)
The dashboard's per-recipe history page (`/recipe/<recipe>`) sources its run list from the local
`/var/lib/cc-ci-runs/*/results.json` artifacts (complete: 308 finished runs; durable; already
bind-mounted read-only), NOT the Drone `…/builds?per_page=100` slice (root cause: that 100-build
window dropped each recipe's older runs out of view after the regall sweep → most recipes showed 1
run). Newest-first by the `results.json` `finished` timestamp (run ids are MIXED numeric + named, so
only a timestamp sort is correct — `int(run_id)` would crash on `m2r-*`/`ab-*`); display-capped at
`HISTORY_CAP=30`. Status derived from the per-stage `results` map (no top-level status field). The
OVERVIEW (`/`) and badges keep their Drone latest-per-recipe source unchanged. Deliberately did NOT
merge Drone live "running" status into history (optional per plan; re-adds the network dependency the
local source removes; overview already shows live status). Retention: 308 parseable runs present, no
trim job observed → adequate; revisit only if a cap is ever needed.

---

## Phase `settings` (2026-06-17) — server settings.toml + SKIP_CANONICALS_FOR_UPGRADE + release-tag-first fallback

- **Settings home = `harness/settings.py` (new), file `/etc/cc-ci/settings.toml` (override `$CCCI_SETTINGS`).**
  No pre-existing cc-ci config module existed to extend (config was scattered `os.environ.get` reads);
  a minimal stdlib-`tomllib` loader is the minimal+extensible mechanism. `_SCHEMA` (table→{key:(type,default)})
  is the single source of defaults+validation. Tracked `settings.toml.example`; live file untracked/operator-
  managed/no-secrets (secrets stay in sops). Default `/etc/cc-ci` chosen over the plan's suggested
  `/srv/cc-ci` (orchestrator-ambiguous): `/etc/cc-ci` is where the harness already runs (`CCCI_REPO`),
  absolute so Drone+sweep read the same file, untracked file survives deploy `git pull`.
- **`SKIP_CANONICALS_FOR_UPGRADE` scope = upgrade BASE only.** Wired into `resolve_upgrade_base`: flag
  true → skip canonical lookup → no-canonical fallback (behaves as if no canonical). Does NOT touch
  canonical *promotion* or the `--quick` warm-reattach — those are separate optimizations; a future
  `SKIP_CANONICAL_SWEEP` / `SKIP_QUICK` could gate them (out of scope here).
- **No-canonical fallback (always-on, §2.C):** newest release TAG `< head` (reuse
  `warm_reconcile.newest_older_version`, the single version-ordering source) → raw main-tip (no prior
  release tag) → skip. Replaces the old jump-straight-to-main-tip path; improves this server too (false
  flag, un-promoted recipes get a real release base).
- **Canonical-present path (incl. samever step-back) preserved byte-for-byte.** With flag false + a
  canonical, behavior is unchanged. The step-back's "no older predecessor → skip" is intentionally NOT
  routed to main-tip (would reintroduce the same-version no-op samever prevents); the §2.C "==head"
  routing is satisfied because the step-back already takes the same release-tag helper as fallback step 1.
- **Validation:** absent/unreadable/malformed-TOML → WARN + all-defaults (cannot crash the harness);
  unknown table/key → warn-and-ignore; present known key of wrong type → raise TypeError (loud typo).
- **OBSERVATION (not this phase's defect):** `scripts/lint.sh` (pinned ruff) reports
  `dashboard/dashboard.py` + `tests/unit/test_dashboard.py` would be reformatted — confirmed pre-existing
  at HEAD f68f1c5, outside the settings diff. Flagged for the dashboard owner / orchestrator; not fixed
  here (narrow scope).

## Warm state is keyed by STACK NAMESPACE, not by recipe (phase redfix, F-redfix-4, 2026-07-09)

**Settled.** `/var/lib/ci-warm/<ns>/` is a slot owned by exactly ONE deployed stack. `ns` comes from
`canonical.canonical_ns(recipe)` for the data-warm canonical and `warmsnap.live_slot(recipe)` for the
live-warm reconciler. A recipe that is both (only `keycloak` today, via `warm.WARM_DOMAINS`) gets
`canon-<recipe>` for its canonical, so the two never share a slot. The canonical's **domain and its slot
derive from the same `canonical_ns()`**, so they cannot drift apart — that coupling is the invariant, not
the `canon-` string. Every slot `<ns>` maps 1:1 to the stack at `warm.stable_domain(<ns>)`.

Rejected alternatives:
- **Skip `seed_canonical` for `WARM_DOMAINS` recipes.** Silently de-enrolls keycloak and re-opens the DoD
  item ("keycloak enrolled … verified green"). The Adversary named this as not acceptable; agreed.
- **Key every slot by the full domain / stack name** (the "purest" model). Correct, but it relocates the
  reconciler's `last_good` for keycloak+traefik and `canonical.json` + `snapshot/` for 15 existing
  canonicals on a live node, requiring a migration shim for a phase whose mandate is to fix red, not
  re-architect. The chosen scheme is the same model applied only where two stacks actually contend, and
  needs no migration (keycloak's canonical had never been seeded).
- **Guard-only (`_assert_slot_not_foreign` alone).** Turns silent destruction into a loud failure but leaves
  keycloak's canonical unable to ever seed. Kept as defence in depth BEHIND the slot separation, not as the
  fix.

Consequence for future enrollments: adding a recipe to `warm.WARM_DOMAINS` automatically namespaces its
canonical on both axes. `prune_stale()`'s "reconciler dirs are never pruned" invariant is now structural
(a `<recipe>/` dir never gains a `canonical.json`) rather than an incidental property that de-enrollment
could falsify.

## 2026-07-10 — Gitea credential rotation + git auth moved to SSH (operator-directed)

Both leaked `autonomic-bot` credentials were rotated and the exposures scrubbed (was B-redfix-8 /
A-redfix-1 / B-redfix-9, previously operator-scope):
- **Account password** rotated (old value now returns HTTP 401). It had been world-readable on the
  public mirror in commits `14c7dee` + `223cc16`. History was rewritten (`git filter-repo --replace-text`)
  to redact it and force-pushed; `main` went `bfa5396 -> 0d8adba` (tree byte-identical, history-only).
  All clones + node `/etc/cc-ci` were reset to the rewritten `main`. Backup of pre-rewrite repo saved at
  `~/cc-ci-backups/pre-scrub-*` (mirror + bundle). **Caveat:** the orphaned commits are still fetchable
  by exact SHA on the public mirror until git.autonomic.zone runs a site-admin GC (value is dead, so inert).
- **`bridge_gitea_token`** (the oauth2 token, sops `secrets.yaml`) rotated: fresh PAT minted, re-encrypted,
  deployed via `nixos-rebuild ?submodules=1`, old token + all 8 other stale PATs revoked. Only one PAT
  remains: `cc-ci-bridge-*` (id 36).

**Git auth is now SSH, not HTTP.** git.autonomic.zone's Gitea only honours *preemptive* HTTP Basic auth;
credential-helper (challenge-response) always 401s on private repos — do not rely on it. Remotes are SSH
(`git@git.autonomic.zone:...`, port 2222 via `~/.ssh/config`); dedicated keys `cc-ci-orchestrator-20260710`
(orchestrator, `~/.ssh/autonomic-bot-gitea-ed25519`) and `cc-ci-node-20260710` (node, `/root/.ssh/...`).
`.gitmodules` secrets URL is SSH. `recipe-mirror-sync.sh` pushes over SSH (no token written into recipe
`.git/config` -> fixes the copytree leak). The bot **password stays in `/srv/cc-ci/.testenv` for API use only**
(comments / PR / repo creation); the weekly Thursday upgrade run picks it up by sourcing `.testenv`.
Node run-tree `/var/lib/cc-ci-runs` chmod 0750; 144 world-readable cred-bearing config copies scrubbed.
