M6: D4 recipe-local discovery + recipe #2 (keycloak, DB-backed) enrolled; M6 CLAIMED

D4 snapshots recipe-shipped tests/ and runs them against the live app. abra -C -o
everywhere + token clone for private mirror PRs. keycloak install green with no
harness surgery (D5). docs/enroll-recipe.md.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

BACKLOG.md

@@ -52,8 +52,13 @@ Two single-writer sections (§6.1): Builder edits only `## Build backlog`; Adver
       Full 3-stage run green: install(2)+upgrade(1)+backup(1) passed; teardown leaves 0 orphans, infra intact.
 
 ### M6 — Recipe-local tests + second recipe
-- [ ] Discover/run recipe-repo tests/; enroll DB-backed recipe #2
-- [ ] Gate: M6 — both green; recipe-local tests merged
+- [x] D4 recipe-local discovery: recipe-shipped tests/ snapshotted post-fetch + run against the live
+      app as a `recipe-local` stage (contract CCCI_BASE_URL/CCCI_APP_DOMAIN). Demo'd via mirror branch
+      recipe-maintainers/custom-html@ci/d4-recipe-local → recipe-local test PASSED against live app.
+- [x] Enroll DB-backed recipe #2 (keycloak + mariadb) via per-recipe tests/keycloak/ only (no harness
+      surgery): install green (realm health + Playwright admin login). docs/enroll-recipe.md written.
+- [x] Gate: M6 — both recipes green (custom-html 3-stage; keycloak install) + recipe-local merged →
+      CLAIMED 2026-05-27. keycloak full 3-stage (DB data survival) folds into the M6.5 breadth ramp.
 
 ### M6.5 — Breadth ramp (recipes 3→6)
 - [ ] Enroll recipes 3–6 covering remaining D10 categories, no harness surgery

DECISIONS.md

@@ -100,6 +100,15 @@ Architecture decisions and dead-ends. One line of rationale each. (§0, §8)
   unique per run, collision-safe across recipes (full recipe in the hash). Human-readable recipe/PR/
   ref context lives in the Drone build params + the PR comment, not the (ephemeral) domain.
 
+- **abra recipe checkout is volatile — harness uses chaos+offline + a tests/ snapshot (M6).** Many
+  abra commands (`app ls`, `secret generate` without flags, version resolution) silently
+  `git checkout <version-tag>` in `~/.abra/recipes/<recipe>`, discarding a PR branch's files. To
+  test the *PR head code* (not a re-resolved tag): (1) `fetch_recipe` clones the mirror branch/ref
+  (private → bot token via per-command `http.extraHeader`, never persisted/logged); (2) all harness
+  abra calls that touch the recipe pass `-C` (chaos: use current checkout) `-o` (offline: no remote
+  fetch); (3) recipe-shipped `tests/` (D4) are **snapshotted to a temp dir right after fetch**, since
+  later abra commands still reset the checkout — the recipe-local stage runs from the snapshot.
+
 ## Risks
 
 - **Disk — RESOLVED 2026-05-26.** Original 8.9 GiB root had only ~3.8 GiB free *and* a hard

JOURNAL.md

@@ -423,3 +423,26 @@ is slow: image pull + JVM + mariadb migration). Teardown clean (0 keyc-* service
 
 **Next:** D4 demo via a mirror shipping committed tests/ (recipe-local run against live app); then
 keycloak upgrade + backup/restore (DB data survival via a realm marker through the admin API).
+
+## 2026-05-27 — M6: D4 recipe-local discovery + recipe #2 enrolled (CLAIMED)
+
+**D4 recipe-local discovery working.** Demo: pushed a committed `tests/test_recipe_local.py` to the
+mirror on branch `recipe-maintainers/custom-html@ci/d4-recipe-local`; ran
+`RECIPE=custom-html SRC=recipe-maintainers/custom-html REF=ci/d4-recipe-local STAGES=install` →
+install 2 passed, then `===== STAGE: recipe-local (D4) =====` ran the recipe-shipped test against
+the LIVE app (CCCI_BASE_URL) → 1 passed. Clean teardown (0 orphans).
+
+**Hard-won abra behaviour (DECISIONS.md):** private mirror clone needs the bot token (per-command
+`http.extraHeader`, not persisted/logged). abra commands (`app ls`, `secret generate`, version
+resolution) silently `git checkout <tag>` the recipe, dropping a PR branch's files — so (1) all
+harness abra calls use `-C -o` (chaos+offline = current checkout, no remote fetch), and (2) D4
+snapshots the recipe's tests/ to a temp dir right after fetch (later abra cmds still reset it).
+Traced the drop step-by-step: app_new ok, deploy ok, but `secret generate` (no flags) and `app ls`
+each reset the checkout.
+
+**Recipe #2 = keycloak** (keycloak + mariadb, DB-backed) install green with only
+`tests/keycloak/recipe_meta.py` + `test_install.py` — **no runner/harness change** (D5). custom-html
+remains 3-stage green (M5). docs/enroll-recipe.md written.
+
+**M6 CLAIMED.** keycloak's full 3-stage (DB data survival via a realm marker) folds into M6.5.
+**Next:** M6.5 — keycloak upgrade/backup, then recipes 3–6 across the remaining D10 categories.

STATUS.md

@@ -1,9 +1,9 @@
 # STATUS — cc-ci Builder
 
-**Phase:** M5 complete & CLAIMED. M0/M1/M2 PASS. M3 gate BLOCKED (Gitea webhook; operator). M4/M5 awaiting verdict.
-Next: M6 (recipe-local tests + DB-backed recipe #2) — and pivot M3 to polling if webhook stays blocked.
-**In-flight:** M6 — discover/run recipe-repo tests/ + enroll a second (DB-backed) recipe.
-**Last updated:** 2026-05-27 (M5 claimed; 3-stage green for custom-html)
+**Phase:** M6 complete & CLAIMED. M0/M1/M2/M4/M5 PASS. M3 gate BLOCKED (Gitea webhook; operator).
+Next: M6.5 (breadth ramp — recipes 3–6 + keycloak full 3-stage), M7, M8. Resolve M3 trigger before M10.
+**In-flight:** M6.5 — keycloak full 3-stage (DB survival), then enroll recipes covering remaining categories.
+**Last updated:** 2026-05-27 (M6 claimed; D4 + recipe #2)
 
 ## Gates
 - **Gate: M0 — CLAIMED, awaiting Adversary** (2026-05-26). Evidence: flake rebuilds cc-ci from repo

docs/enroll-recipe.md

@@ -0,0 +1,54 @@
+# Enrolling a recipe under cc-ci (D5)
+
+Adding a recipe is a small, repeatable, **no-harness-surgery** operation:
+
+## 1. Make the recipe available on the mirror
+
+Recipes under test live on the private mirror `git.autonomic.zone/recipe-maintainers/<recipe>`,
+synced from upstream `git.coopcloud.tech`. If not yet mirrored, mirror it (abra fetch + push to the
+org) — see the recipe mirror+PR flow (plan §4.1). A recipe may ship its own `tests/` dir in its repo;
+those are discovered and run against the live app (D4 — see below).
+
+## 2. Add the per-recipe test tree in this repo
+
+```
+tests/<recipe>/
+├── recipe_meta.py      # optional per-recipe harness config (see below)
+├── test_install.py     # install-stage assertions (health + Playwright)
+├── test_upgrade.py     # upgrade-stage assertions (data survives)
+└── test_backup.py      # backup→mutate→restore assertions
+```
+
+Copy from an existing recipe (e.g. `tests/custom-html/` for a simple app, `tests/keycloak/` for a
+DB-backed one). The shared fixtures live in `tests/conftest.py` + `runner/harness/` — **do not edit
+them to add a recipe**; instead set per-recipe config in `recipe_meta.py`:
+
+```python
+HEALTH_PATH = "/realms/master"   # path that returns a healthy status (default "/")
+HEALTH_OK = (200,)               # acceptable status codes (default 200/301/302)
+DEPLOY_TIMEOUT = 600             # seconds for services to converge (default 600)
+HTTP_TIMEOUT = 600               # seconds for the app to answer (default 300)
+```
+
+The test files use the fixtures: `deployed_app` (install), `deployed` (function-scoped), and the
+`harness.lifecycle` helpers (`http_get`, `http_body`, `exec_in_app`, `upgrade_app`, `backup_app`,
+`restore_app`, `previous_version`). The harness forces `LETS_ENCRYPT_ENV=""` (no ACME) and a unique
+short domain per run, and guarantees teardown.
+
+## 3. Recipe-local tests (D4)
+
+If the recipe's own repo contains `tests/test_*.py`, the runner snapshots them right after fetch and
+runs them against the **live deployment** as a `recipe-local` stage. Contract: those tests receive
+env `CCCI_BASE_URL` (e.g. `https://<app>.ci.commoninternet.net/`) and `CCCI_APP_DOMAIN`.
+
+## 4. Register the trigger webhook
+
+Add the per-repo Gitea webhook so `!testme` on a PR starts a run (see the bridge / runbook). Then
+`!testme` on a PR runs install/upgrade/backup + any recipe-local tests, and reports back to the PR.
+
+## Run locally
+
+```sh
+RECIPE=<recipe> PR=<n> REF=<sha-or-branch> SRC=recipe-maintainers/<recipe> \
+  STAGES=install,upgrade,backup cc-ci-run runner/run_recipe_ci.py
+```

runner/harness/abra.py

@@ -49,7 +49,9 @@ def app_new(recipe: str, domain: str, server: str = "default", version: Optional
     args = ["app", "new", recipe]
     if version:
         args.append(version)
-    args += ["-s", server, "-D", domain, "-n"]
+    # -C (chaos): deploy the recipe AT THE CURRENT CHECKOUT (the PR head under test), not a
+    # re-resolved version tag. -o (offline): don't fetch tags from the (private) remote.
+    args += ["-s", server, "-D", domain, "-C", "-o", "-n"]
     if secrets:
         args.append("-S")
     _run(args)
@@ -78,12 +80,15 @@ def env_set(domain: str, key: str, value: str) -> None:
 
 def secret_generate(domain: str, timeout: int = 300) -> None:
     # -m avoids the TTY/table (ioctl) path; output (which contains the generated values) is
-    # captured by _run and never logged. check=False: recipes with no secrets are a no-op.
-    _run(["app", "secret", "generate", domain, "--all", "-m", "-n"], timeout=timeout, check=False)
+    # captured by _run and never logged. -C -o keep the recipe at the PR checkout (without -o it
+    # re-resolves to a version tag, dropping the PR's files incl. tests/). check=False: recipes with
+    # no secrets are a no-op.
+    _run(["app", "secret", "generate", domain, "--all", "-m", "-C", "-o", "-n"],
+         timeout=timeout, check=False)
 
 
 def deploy(domain: str, chaos: bool = True, timeout: int = 900) -> None:
-    args = ["app", "deploy", domain, "-n"]
+    args = ["app", "deploy", domain, "-o", "-n"]
     if chaos:
         args.append("-C")
     _run(args, timeout=timeout)

runner/run_recipe_ci.py

@@ -18,8 +18,10 @@ from __future__ import annotations
 
 import glob
 import os
+import shutil
 import subprocess
 import sys
+import tempfile
 
 ROOT = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
 sys.path.insert(0, os.path.join(ROOT, "runner"))
@@ -31,17 +33,29 @@ STAGE_FILES = {
 }
 
 
+def _gitea_token() -> str | None:
+    tok = os.environ.get("GITEA_TOKEN")
+    if not tok and os.path.exists("/run/secrets/bridge_gitea_token"):
+        tok = open("/run/secrets/bridge_gitea_token").read().strip()
+    return tok or None
+
+
 def fetch_recipe(recipe: str, ref: str | None, src: str | None) -> None:
     """Make the recipe available at the code under test. If SRC+REF point at the mirror PR,
-    clone it at that ref; otherwise fetch the catalogue copy."""
+    clone it at that ref; otherwise fetch the catalogue copy. Private mirror repos need the bot
+    token — passed via a per-command http.extraHeader (not persisted in .git/config, not printed)."""
     recipes_dir = os.path.expanduser("~/.abra/recipes")
     os.makedirs(recipes_dir, exist_ok=True)
     dest = os.path.join(recipes_dir, recipe)
     if src and ref:
         url = f"https://git.autonomic.zone/{src}.git"
+        git = ["git"]
+        tok = _gitea_token()
+        if tok:
+            git += ["-c", f"http.extraHeader=Authorization: token {tok}"]
         subprocess.run(["rm", "-rf", dest], check=False)
-        subprocess.run(["git", "clone", "--quiet", url, dest], check=True)
-        subprocess.run(["git", "-C", dest, "checkout", "--quiet", ref], check=True)
+        subprocess.run([*git, "clone", "--quiet", url, dest], check=True)
+        subprocess.run([*git, "-C", dest, "checkout", "--quiet", ref], check=True)
     else:
         subprocess.run(["abra", "recipe", "fetch", recipe, "-n"], check=True)
 
@@ -57,6 +71,9 @@ def main() -> int:
 
     print(f"== cc-ci run: recipe={recipe} ref={ref} pr={os.environ.get('PR', '0')} stages={stages}")
     fetch_recipe(recipe, ref, src)
+    # Snapshot any recipe-shipped tests/ NOW — later abra commands (app ls, deploy, …) re-checkout
+    # the recipe to a version tag, which would drop the PR's tests/. (D4)
+    local_tests = snapshot_recipe_tests(recipe)
 
     test_dir = os.path.join(ROOT, "tests", recipe)
     overall = 0
@@ -79,7 +96,7 @@ def main() -> int:
     # D4: recipe-local tests. If the recipe repo ships a tests/ dir, deploy the app and run those
     # tests against the LIVE deployment (contract: CCCI_BASE_URL + CCCI_APP_DOMAIN env), merging the
     # result into this run as another reported stage. Teardown is guaranteed.
-    rc = run_recipe_local(recipe)
+    rc = run_recipe_local(recipe, local_tests)
     if rc is not None:
         ran += 1
         if rc != 0:
@@ -91,9 +108,20 @@ def main() -> int:
     return overall
 
 
-def run_recipe_local(recipe: str) -> int | None:
-    local_dir = os.path.expanduser(f"~/.abra/recipes/{recipe}/tests")
-    if not os.path.isdir(local_dir) or not glob.glob(os.path.join(local_dir, "test_*.py")):
+def snapshot_recipe_tests(recipe: str) -> str | None:
+    """Copy the recipe-shipped tests/ to a stable temp dir, immune to abra re-checking-out the
+    recipe to a version tag during the run. Returns the snapshot path, or None if no tests/."""
+    src = os.path.expanduser(f"~/.abra/recipes/{recipe}/tests")
+    if not os.path.isdir(src) or not glob.glob(os.path.join(src, "test_*.py")):
+        return None
+    dst = os.path.join(tempfile.gettempdir(), f"ccci-recipe-tests-{recipe}")
+    shutil.rmtree(dst, ignore_errors=True)
+    shutil.copytree(src, dst)
+    return dst
+
+
+def run_recipe_local(recipe: str, local_tests: str | None) -> int | None:
+    if not local_tests:
         return None  # recipe ships no tests/ — D4 is a no-op for it
     print("\n===== STAGE: recipe-local (D4) =====", flush=True)
     domain = naming.app_domain(recipe, os.environ.get("PR", "0"), os.environ.get("REF"))
@@ -102,7 +130,7 @@ def run_recipe_local(recipe: str) -> int | None:
         lifecycle.deploy_app(recipe, domain, version=os.environ.get("VERSION") or None)
         lifecycle.wait_healthy(domain)
         env = dict(os.environ, CCCI_APP_DOMAIN=domain, CCCI_BASE_URL=f"https://{domain}")
-        return subprocess.call([sys.executable, "-m", "pytest", "-v", "-rA", local_dir],
+        return subprocess.call([sys.executable, "-m", "pytest", "-v", "-rA", local_tests],
                                cwd=ROOT, env=env)
     finally:
         lifecycle.teardown_app(domain, verify=False)