diff --git a/docs/recipe-customization.md b/docs/recipe-customization.md index c772044..ef2fea9 100644 --- a/docs/recipe-customization.md +++ b/docs/recipe-customization.md @@ -115,8 +115,8 @@ _This table is GENERATED from the `runner/harness/meta.py` KEYS registry by `scr | `HEALTH_OK` | `tuple[int]` | `(200, 301, 302)` | Acceptable HTTP status codes for health. | | `DEPLOY_TIMEOUT` | `int` | `600` | Max seconds to wait for swarm convergence per deploy. | | `HTTP_TIMEOUT` | `int` | `300` | Max seconds to wait for HTTP health after convergence. | -| `BACKUP_CAPABLE` | `bool` | `None` | Override the backup-tier capability auto-detect (compose `backupbot.backup` labels). `False` forces an intentional skip of the backup/restore rung; `True` forces the tier on; unset = auto-detect. | -| `EXPECTED_NA` | `dict` | `None` | Declare a non-run rung an INTENTIONAL skip: `{rung: reason}`. The level climbs past an intentional skip; an undeclared non-run rung is *unverified* and blocks the level above it (phase lvl5; classification table in `machine-docs/DECISIONS.md`). Never overrides an exercised pass/fail; the `lint` rung has no escape hatch. | +| `BACKUP_CAPABLE` | `bool` | `None` | Override the backup-tier capability auto-detect (compose `backupbot.backup` labels). `False` forces N/A; `True` forces the tier on; unset = auto-detect. | +| `EXPECTED_NA` | `dict` | `None` | Declare an N/A rung intentional: `{rung: reason}`. The cap stands either way; only the report wording changes. | | `READY_PROBE` | `hook` | `None` | Callable `(ctx) -> [probe, ...]` returning extra readiness probes, run after install AND after upgrade: HTTP `{host, path, ok}` or TCP `{tcp_host, tcp_port, stable}`. | | `UPGRADE_BASE_VERSION` | `str` | `None` | Exact published tag overriding the upgrade tier's base (default: `recipe_versions[-2]`). | | `BACKUP_VERIFY` | `hook` | `None` | Callable `(ctx) -> bool` post-backup data-capture check; `False` re-runs the backup (truncated-dump race guard), retried up to 3 attempts. | diff --git a/docs/results-ux.md b/docs/results-ux.md index 6ca980e..270daba 100644 --- a/docs/results-ux.md +++ b/docs/results-ux.md @@ -10,9 +10,12 @@ It is the R8 reference for Phase 3 (`plan-phase3-results-ux.md`). --- -## 1. The level ladder (phase lvl5 semantics, operator-decided 2026-06-11) +## 1. The level ladder (R1) -Every run earns a single integer **level 0–5** over the FIVE essential rungs: +Every run earns a single integer **level 0–6**. The ladder is cumulative with **YunoHost +gap-caps-the-level** semantics: you earn level `L` only if **every rung 1..L was a clean PASS**. The +first rung that is not a clean PASS — a real **FAIL** *or* genuinely **N/A** for this recipe — stops +the climb, and `level_cap_reason` records which rung and why. | Level | Rung | Earned when | |------:|------|-------------| @@ -21,52 +24,42 @@ Every run earns a single integer **level 0–5** over the FIVE essential rungs: | **L2** | upgrade | previous published version → PR/latest, stays healthy, data intact. | | **L3** | backup/restore | seeded data survives backup → wipe → restore. | | **L4** | functional | the recipe-specific functional tests pass. | -| **L5** | lint | `abra recipe lint` passes against the exact ref under test. | +| **L5** | integration | SSO/OIDC + cross-app integration tests pass. | +| **L6** | recipe-local | the recipe repo's own `tests/` (D4) pass and are merged. | -Each rung has one of FOUR statuses, and the level is: +**N/A caps, fairly.** A rung that does not apply to a recipe (only one published version → no +upgrade; not backup-capable; no SSO/integration surface; no recipe-local tests) is **N/A**, which +caps the climb at the rung below it with a recorded reason — it is *not* counted as a failure. This is +the only fair reading of "a missing lower rung caps the level": e.g. a recipe with **no integration +surface caps at L4 by definition**, shown as `level_cap_reason = "L5 integration … N/A"`. A stateless +app whose functional tests pass but which cannot be backed up is honestly capped at **L2** (`"L3 +backup/restore … N/A"`) rather than shown as L4 — understating is safe; overstating is forbidden. - level = the highest rung that PASSED, where every rung below it is "pass" or an intentional skip - -- **pass / fail** — the rung was exercised. A FAIL blocks: no rung above it counts, however green. -- **skip (intentional)** — the rung *genuinely does not apply*, from a declared or structural fact: - not backup-capable (declared), only one published version (no upgrade target), or a declared - `EXPECTED_NA`. Intentional skips are **climbed past** — a stateless recipe with passing - functional tests and a clean lint reaches **L5**, not the old "capped at 2". -- **unver (unverified)** — the rung *should* have run but didn't: infra error, missing tool, - harness exception, prior-stage abort, timeout. **The level cannot rise above an unverified - rung** — it blocks exactly like a fail (we never claim what we didn't check). Anything - unclassifiable defaults to unver (conservative). - -There is **no capping concept** (no `cap_reason`, no `capped`): the per-rung table -(✔ / ✘ / intentional-skip / unverified) on the card and in `results.json.rungs` is the sole -carrier of "why isn't this level higher". Worked examples: - -- install ✔, upgrade ✘, backup ✔, functional ✔, lint ✔ → **level 1** (fail blocks). -- install ✔, upgrade ✔, backup skip (not capable), functional ✔, lint ✔ → **level 5**. -- install ✔, upgrade ✔, backup unver (harness error), functional ✔, lint ✔ → **level 2**. -- all four ✔, lint unver (abra missing) → **level 4** (an unverified top rung isn't earned). - -Integration (SSO/OIDC + cross-app) and recipe-local tests are **optional capabilities**, not -rungs — they never affect the level (SSO remains enforced for the run VERDICT). +Worked examples (real runs): +- `uptime-kuma` — install+upgrade+backup+restore+functional all pass, no SSO surface → **L4** + (`cap = "L5 integration (SSO/OIDC + cross-app) N/A"`). +- `custom-html-tiny` — stateless, not backup-capable: install+upgrade pass, backup/restore N/A → + **L2** (`cap = "L3 backup/restore (data integrity) N/A"`). ### How tiers map to rungs (the translation layer) `run_recipe_ci.py` holds the run's per-tier results (`install/upgrade/backup/restore/custom`) + -structural signals; `runner/harness/results.py::derive_rungs` maps them to the rung-status dict -that `runner/harness/level.py::compute_level` scores. The full intentional-vs-unintentional -classification table for every N/A source is in `machine-docs/DECISIONS.md` (phase lvl5). Summary: +deps/SSO signals; `runner/harness/results.py::derive_rungs` maps them to the rung-status dict that +`runner/harness/level.py::compute_level` scores. The mapping (also in `DECISIONS.md`, Phase 3): -- **install** ← install tier (pass/fail; a non-run is unver — install always applies). -- **upgrade** ← upgrade tier; tier skipped with no upgrade target (single published version, - structural) → skip; declared `EXPECTED_NA` → skip; otherwise unver. +- **install** ← install tier (pass/fail). +- **upgrade** ← upgrade tier; `skip` → **na** (only one published version). - **backup_restore** ← backup AND restore tiers both pass → pass; either fail → fail; not - backup-capable (structural/declared) → skip; unverified-while-capable → unver. -- **functional** ← the custom tier; a custom failure conservatively fails this rung; no custom - tests is a coverage GAP → unver, unless declared `EXPECTED_NA["functional"]` → skip. -- **lint** ← the lint executor (`runner/harness/lint.py`): `abra recipe lint` on a pristine - scratch clone of the run's recipe tree at the exact tested sha, 60s hard budget, full output in - the run artifact `lint.txt`. pass/fail only — when lint can't run the rung is **unver** (never - a silent pass, never an intentional skip). Lint never changes the run verdict. + backup-capable → **na**. +- **functional** ← the custom tier minus its SSO tests; a custom failure conservatively fails this + rung (we don't split functional-vs-SSO failure → never inflate); no custom tests → **na**. +- **integration** ← applies only if the recipe declares deps; pass iff deps wired and SSO verified and + custom didn't fail; recipes with no declared deps → **na** (the "caps at L4" rule). +- **recipe_local** ← the recipe repo's own `tests/` (discovery source `repo-local`) ran and passed; + none present → **na**. + +The pure scorer is exhaustively unit-tested + fuzz-verified (all 729 rung combinations: level == +count of leading consecutive passes, zero inflation). ### Invariant flags (shown, not climbed) @@ -84,29 +77,19 @@ build number, or the run's unique app domain for a hand-run). Schema: ```json { - "schema": 2, "run_id": "...", "recipe": "...", "version": "...", "pr": "...", "ref": "...", + "schema": 1, "run_id": "...", "recipe": "...", "version": "...", "pr": "...", "ref": "...", "finished": 0.0, - "level": 5, - "rungs": {"install":"pass","upgrade":"pass","backup_restore":"skip","functional":"pass", - "lint":"pass"}, - "lint": {"status":"pass","detail":"","rules_failed":[]}, - "skips": {"intentional": {"backup_restore": "not backup-capable (no backupbot labels / declared)"}, - "unintentional": []}, + "level": 4, "level_cap_reason": "L5 integration (SSO/OIDC + cross-app) N/A", + "rungs": {"install":"pass","upgrade":"pass","backup_restore":"pass","functional":"pass", + "integration":"na","recipe_local":"na"}, "stages": [{"name":"install","status":"pass", "tests":[{"name":"test_serving","status":"pass","ms":168,"source":"generic"}]}], - "results": {"install":"pass","upgrade":"pass","backup":"skip","restore":"skip","custom":"pass"}, + "results": {"install":"pass","upgrade":"pass","backup":"pass","restore":"pass","custom":"pass"}, "flags": {"clean_teardown": true, "no_secret_leak": true}, "screenshot": "screenshot.png", "summary_card": "summary.png" } ``` -`rungs` carries the four-status vocabulary above; `skips.intentional` maps each intentionally -skipped rung to its (declared or structural) reason and `skips.unintentional` lists the -unverified rungs. `lint` carries the L5 rung outcome + failing rule ids; the full -`abra recipe lint` output is served at `/runs//lint.txt`. Pre-lvl5 artifacts -(`"schema": 1`, 4-rung ladder, `level_cap_reason`/`level_cap_rung` present, `"na"` statuses) -are still rendered as-is by the dashboard/card — their stored level is never recomputed. - Assembly is **best-effort**: a failure to build/write `results.json` is logged but never changes the run's exit code (cosmetics never block the pipeline, R7).