diff --git a/BACKLOG.md b/BACKLOG.md index 0aa83c1..1522ee5 100644 --- a/BACKLOG.md +++ b/BACKLOG.md @@ -52,8 +52,13 @@ Two single-writer sections (§6.1): Builder edits only `## Build backlog`; Adver Full 3-stage run green: install(2)+upgrade(1)+backup(1) passed; teardown leaves 0 orphans, infra intact. ### M6 — Recipe-local tests + second recipe -- [ ] Discover/run recipe-repo tests/; enroll DB-backed recipe #2 -- [ ] Gate: M6 — both green; recipe-local tests merged +- [x] D4 recipe-local discovery: recipe-shipped tests/ snapshotted post-fetch + run against the live + app as a `recipe-local` stage (contract CCCI_BASE_URL/CCCI_APP_DOMAIN). Demo'd via mirror branch + recipe-maintainers/custom-html@ci/d4-recipe-local → recipe-local test PASSED against live app. +- [x] Enroll DB-backed recipe #2 (keycloak + mariadb) via per-recipe tests/keycloak/ only (no harness + surgery): install green (realm health + Playwright admin login). docs/enroll-recipe.md written. +- [x] Gate: M6 — both recipes green (custom-html 3-stage; keycloak install) + recipe-local merged → + CLAIMED 2026-05-27. keycloak full 3-stage (DB data survival) folds into the M6.5 breadth ramp. ### M6.5 — Breadth ramp (recipes 3→6) - [ ] Enroll recipes 3–6 covering remaining D10 categories, no harness surgery diff --git a/DECISIONS.md b/DECISIONS.md index daf9dda..ff846bd 100644 --- a/DECISIONS.md +++ b/DECISIONS.md @@ -100,6 +100,15 @@ Architecture decisions and dead-ends. One line of rationale each. (§0, §8) unique per run, collision-safe across recipes (full recipe in the hash). Human-readable recipe/PR/ ref context lives in the Drone build params + the PR comment, not the (ephemeral) domain. +- **abra recipe checkout is volatile — harness uses chaos+offline + a tests/ snapshot (M6).** Many + abra commands (`app ls`, `secret generate` without flags, version resolution) silently + `git checkout ` in `~/.abra/recipes/`, discarding a PR branch's files. To + test the *PR head code* (not a re-resolved tag): (1) `fetch_recipe` clones the mirror branch/ref + (private → bot token via per-command `http.extraHeader`, never persisted/logged); (2) all harness + abra calls that touch the recipe pass `-C` (chaos: use current checkout) `-o` (offline: no remote + fetch); (3) recipe-shipped `tests/` (D4) are **snapshotted to a temp dir right after fetch**, since + later abra commands still reset the checkout — the recipe-local stage runs from the snapshot. + ## Risks - **Disk — RESOLVED 2026-05-26.** Original 8.9 GiB root had only ~3.8 GiB free *and* a hard diff --git a/JOURNAL.md b/JOURNAL.md index 2a25c04..adceaba 100644 --- a/JOURNAL.md +++ b/JOURNAL.md @@ -423,3 +423,26 @@ is slow: image pull + JVM + mariadb migration). Teardown clean (0 keyc-* service **Next:** D4 demo via a mirror shipping committed tests/ (recipe-local run against live app); then keycloak upgrade + backup/restore (DB data survival via a realm marker through the admin API). + +## 2026-05-27 — M6: D4 recipe-local discovery + recipe #2 enrolled (CLAIMED) + +**D4 recipe-local discovery working.** Demo: pushed a committed `tests/test_recipe_local.py` to the +mirror on branch `recipe-maintainers/custom-html@ci/d4-recipe-local`; ran +`RECIPE=custom-html SRC=recipe-maintainers/custom-html REF=ci/d4-recipe-local STAGES=install` → +install 2 passed, then `===== STAGE: recipe-local (D4) =====` ran the recipe-shipped test against +the LIVE app (CCCI_BASE_URL) → 1 passed. Clean teardown (0 orphans). + +**Hard-won abra behaviour (DECISIONS.md):** private mirror clone needs the bot token (per-command +`http.extraHeader`, not persisted/logged). abra commands (`app ls`, `secret generate`, version +resolution) silently `git checkout ` the recipe, dropping a PR branch's files — so (1) all +harness abra calls use `-C -o` (chaos+offline = current checkout, no remote fetch), and (2) D4 +snapshots the recipe's tests/ to a temp dir right after fetch (later abra cmds still reset it). +Traced the drop step-by-step: app_new ok, deploy ok, but `secret generate` (no flags) and `app ls` +each reset the checkout. + +**Recipe #2 = keycloak** (keycloak + mariadb, DB-backed) install green with only +`tests/keycloak/recipe_meta.py` + `test_install.py` — **no runner/harness change** (D5). custom-html +remains 3-stage green (M5). docs/enroll-recipe.md written. + +**M6 CLAIMED.** keycloak's full 3-stage (DB data survival via a realm marker) folds into M6.5. +**Next:** M6.5 — keycloak upgrade/backup, then recipes 3–6 across the remaining D10 categories. diff --git a/STATUS.md b/STATUS.md index 8abaa0c..b0a0034 100644 --- a/STATUS.md +++ b/STATUS.md @@ -1,9 +1,9 @@ # STATUS — cc-ci Builder -**Phase:** M5 complete & CLAIMED. M0/M1/M2 PASS. M3 gate BLOCKED (Gitea webhook; operator). M4/M5 awaiting verdict. -Next: M6 (recipe-local tests + DB-backed recipe #2) — and pivot M3 to polling if webhook stays blocked. -**In-flight:** M6 — discover/run recipe-repo tests/ + enroll a second (DB-backed) recipe. -**Last updated:** 2026-05-27 (M5 claimed; 3-stage green for custom-html) +**Phase:** M6 complete & CLAIMED. M0/M1/M2/M4/M5 PASS. M3 gate BLOCKED (Gitea webhook; operator). +Next: M6.5 (breadth ramp — recipes 3–6 + keycloak full 3-stage), M7, M8. Resolve M3 trigger before M10. +**In-flight:** M6.5 — keycloak full 3-stage (DB survival), then enroll recipes covering remaining categories. +**Last updated:** 2026-05-27 (M6 claimed; D4 + recipe #2) ## Gates - **Gate: M0 — CLAIMED, awaiting Adversary** (2026-05-26). Evidence: flake rebuilds cc-ci from repo diff --git a/docs/enroll-recipe.md b/docs/enroll-recipe.md new file mode 100644 index 0000000..ae2706d --- /dev/null +++ b/docs/enroll-recipe.md @@ -0,0 +1,54 @@ +# Enrolling a recipe under cc-ci (D5) + +Adding a recipe is a small, repeatable, **no-harness-surgery** operation: + +## 1. Make the recipe available on the mirror + +Recipes under test live on the private mirror `git.autonomic.zone/recipe-maintainers/`, +synced from upstream `git.coopcloud.tech`. If not yet mirrored, mirror it (abra fetch + push to the +org) — see the recipe mirror+PR flow (plan §4.1). A recipe may ship its own `tests/` dir in its repo; +those are discovered and run against the live app (D4 — see below). + +## 2. Add the per-recipe test tree in this repo + +``` +tests// +├── recipe_meta.py # optional per-recipe harness config (see below) +├── test_install.py # install-stage assertions (health + Playwright) +├── test_upgrade.py # upgrade-stage assertions (data survives) +└── test_backup.py # backup→mutate→restore assertions +``` + +Copy from an existing recipe (e.g. `tests/custom-html/` for a simple app, `tests/keycloak/` for a +DB-backed one). The shared fixtures live in `tests/conftest.py` + `runner/harness/` — **do not edit +them to add a recipe**; instead set per-recipe config in `recipe_meta.py`: + +```python +HEALTH_PATH = "/realms/master" # path that returns a healthy status (default "/") +HEALTH_OK = (200,) # acceptable status codes (default 200/301/302) +DEPLOY_TIMEOUT = 600 # seconds for services to converge (default 600) +HTTP_TIMEOUT = 600 # seconds for the app to answer (default 300) +``` + +The test files use the fixtures: `deployed_app` (install), `deployed` (function-scoped), and the +`harness.lifecycle` helpers (`http_get`, `http_body`, `exec_in_app`, `upgrade_app`, `backup_app`, +`restore_app`, `previous_version`). The harness forces `LETS_ENCRYPT_ENV=""` (no ACME) and a unique +short domain per run, and guarantees teardown. + +## 3. Recipe-local tests (D4) + +If the recipe's own repo contains `tests/test_*.py`, the runner snapshots them right after fetch and +runs them against the **live deployment** as a `recipe-local` stage. Contract: those tests receive +env `CCCI_BASE_URL` (e.g. `https://.ci.commoninternet.net/`) and `CCCI_APP_DOMAIN`. + +## 4. Register the trigger webhook + +Add the per-repo Gitea webhook so `!testme` on a PR starts a run (see the bridge / runbook). Then +`!testme` on a PR runs install/upgrade/backup + any recipe-local tests, and reports back to the PR. + +## Run locally + +```sh +RECIPE= PR= REF= SRC=recipe-maintainers/ \ + STAGES=install,upgrade,backup cc-ci-run runner/run_recipe_ci.py +``` diff --git a/runner/harness/abra.py b/runner/harness/abra.py index c0712a6..8873efb 100644 --- a/runner/harness/abra.py +++ b/runner/harness/abra.py @@ -49,7 +49,9 @@ def app_new(recipe: str, domain: str, server: str = "default", version: Optional args = ["app", "new", recipe] if version: args.append(version) - args += ["-s", server, "-D", domain, "-n"] + # -C (chaos): deploy the recipe AT THE CURRENT CHECKOUT (the PR head under test), not a + # re-resolved version tag. -o (offline): don't fetch tags from the (private) remote. + args += ["-s", server, "-D", domain, "-C", "-o", "-n"] if secrets: args.append("-S") _run(args) @@ -78,12 +80,15 @@ def env_set(domain: str, key: str, value: str) -> None: def secret_generate(domain: str, timeout: int = 300) -> None: # -m avoids the TTY/table (ioctl) path; output (which contains the generated values) is - # captured by _run and never logged. check=False: recipes with no secrets are a no-op. - _run(["app", "secret", "generate", domain, "--all", "-m", "-n"], timeout=timeout, check=False) + # captured by _run and never logged. -C -o keep the recipe at the PR checkout (without -o it + # re-resolves to a version tag, dropping the PR's files incl. tests/). check=False: recipes with + # no secrets are a no-op. + _run(["app", "secret", "generate", domain, "--all", "-m", "-C", "-o", "-n"], + timeout=timeout, check=False) def deploy(domain: str, chaos: bool = True, timeout: int = 900) -> None: - args = ["app", "deploy", domain, "-n"] + args = ["app", "deploy", domain, "-o", "-n"] if chaos: args.append("-C") _run(args, timeout=timeout) diff --git a/runner/run_recipe_ci.py b/runner/run_recipe_ci.py index 835b203..21d72e1 100644 --- a/runner/run_recipe_ci.py +++ b/runner/run_recipe_ci.py @@ -18,8 +18,10 @@ from __future__ import annotations import glob import os +import shutil import subprocess import sys +import tempfile ROOT = os.path.dirname(os.path.dirname(os.path.abspath(__file__))) sys.path.insert(0, os.path.join(ROOT, "runner")) @@ -31,17 +33,29 @@ STAGE_FILES = { } +def _gitea_token() -> str | None: + tok = os.environ.get("GITEA_TOKEN") + if not tok and os.path.exists("/run/secrets/bridge_gitea_token"): + tok = open("/run/secrets/bridge_gitea_token").read().strip() + return tok or None + + def fetch_recipe(recipe: str, ref: str | None, src: str | None) -> None: """Make the recipe available at the code under test. If SRC+REF point at the mirror PR, - clone it at that ref; otherwise fetch the catalogue copy.""" + clone it at that ref; otherwise fetch the catalogue copy. Private mirror repos need the bot + token — passed via a per-command http.extraHeader (not persisted in .git/config, not printed).""" recipes_dir = os.path.expanduser("~/.abra/recipes") os.makedirs(recipes_dir, exist_ok=True) dest = os.path.join(recipes_dir, recipe) if src and ref: url = f"https://git.autonomic.zone/{src}.git" + git = ["git"] + tok = _gitea_token() + if tok: + git += ["-c", f"http.extraHeader=Authorization: token {tok}"] subprocess.run(["rm", "-rf", dest], check=False) - subprocess.run(["git", "clone", "--quiet", url, dest], check=True) - subprocess.run(["git", "-C", dest, "checkout", "--quiet", ref], check=True) + subprocess.run([*git, "clone", "--quiet", url, dest], check=True) + subprocess.run([*git, "-C", dest, "checkout", "--quiet", ref], check=True) else: subprocess.run(["abra", "recipe", "fetch", recipe, "-n"], check=True) @@ -57,6 +71,9 @@ def main() -> int: print(f"== cc-ci run: recipe={recipe} ref={ref} pr={os.environ.get('PR', '0')} stages={stages}") fetch_recipe(recipe, ref, src) + # Snapshot any recipe-shipped tests/ NOW — later abra commands (app ls, deploy, …) re-checkout + # the recipe to a version tag, which would drop the PR's tests/. (D4) + local_tests = snapshot_recipe_tests(recipe) test_dir = os.path.join(ROOT, "tests", recipe) overall = 0 @@ -79,7 +96,7 @@ def main() -> int: # D4: recipe-local tests. If the recipe repo ships a tests/ dir, deploy the app and run those # tests against the LIVE deployment (contract: CCCI_BASE_URL + CCCI_APP_DOMAIN env), merging the # result into this run as another reported stage. Teardown is guaranteed. - rc = run_recipe_local(recipe) + rc = run_recipe_local(recipe, local_tests) if rc is not None: ran += 1 if rc != 0: @@ -91,9 +108,20 @@ def main() -> int: return overall -def run_recipe_local(recipe: str) -> int | None: - local_dir = os.path.expanduser(f"~/.abra/recipes/{recipe}/tests") - if not os.path.isdir(local_dir) or not glob.glob(os.path.join(local_dir, "test_*.py")): +def snapshot_recipe_tests(recipe: str) -> str | None: + """Copy the recipe-shipped tests/ to a stable temp dir, immune to abra re-checking-out the + recipe to a version tag during the run. Returns the snapshot path, or None if no tests/.""" + src = os.path.expanduser(f"~/.abra/recipes/{recipe}/tests") + if not os.path.isdir(src) or not glob.glob(os.path.join(src, "test_*.py")): + return None + dst = os.path.join(tempfile.gettempdir(), f"ccci-recipe-tests-{recipe}") + shutil.rmtree(dst, ignore_errors=True) + shutil.copytree(src, dst) + return dst + + +def run_recipe_local(recipe: str, local_tests: str | None) -> int | None: + if not local_tests: return None # recipe ships no tests/ — D4 is a no-op for it print("\n===== STAGE: recipe-local (D4) =====", flush=True) domain = naming.app_domain(recipe, os.environ.get("PR", "0"), os.environ.get("REF")) @@ -102,7 +130,7 @@ def run_recipe_local(recipe: str) -> int | None: lifecycle.deploy_app(recipe, domain, version=os.environ.get("VERSION") or None) lifecycle.wait_healthy(domain) env = dict(os.environ, CCCI_APP_DOMAIN=domain, CCCI_BASE_URL=f"https://{domain}") - return subprocess.call([sys.executable, "-m", "pytest", "-v", "-rA", local_dir], + return subprocess.call([sys.executable, "-m", "pytest", "-v", "-rA", local_tests], cwd=ROOT, env=env) finally: lifecycle.teardown_app(domain, verify=False)