refactor(level): four essential rungs only — integration & recipe-local are optional
Some checks failed
continuous-integration/drone/push Build is failing

Per operator: the level ladder is now the FOUR essential rungs every recipe is
held to — install, upgrade (essential), backup/restore, functional (top = L4).
Integration (SSO/OIDC) and recipe-local are OPTIONAL capabilities: they no longer
appear as level rungs or skip rows and never cap the level. SSO is still enforced
for the run VERDICT (unchanged in run_recipe_ci.py); it just doesn't affect the
level. derive_rungs simplified accordingly (drops declared/deps/sso/repo-local
inputs). custom-html-tiny's EXPECTED_NA is back to just backup_restore.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
autonomic-bot
2026-06-09 02:55:47 +00:00
parent 3980340727
commit 46e2cdb93e
9 changed files with 63 additions and 204 deletions

View File

@ -141,14 +141,12 @@ def _stage_rows(stages: list[dict]) -> str:
return "\n".join(rows) or '<tr><td colspan="3">no stages</td></tr>' return "\n".join(rows) or '<tr><td colspan="3">no stages</td></tr>'
# Friendly rung labels for the skip rows. # Friendly rung labels for the skip rows (the four essential rungs).
RUNG_LABEL = { RUNG_LABEL = {
"install": "install", "install": "install",
"upgrade": "upgrade", "upgrade": "upgrade",
"backup_restore": "backup/restore", "backup_restore": "backup/restore",
"functional": "functional", "functional": "functional",
"integration": "integration",
"recipe_local": "recipe-local",
} }
SKIP_GREEN = "#57ab5a" # muted green — an intentional skip reads like a pass (but labelled, never inflating) SKIP_GREEN = "#57ab5a" # muted green — an intentional skip reads like a pass (but labelled, never inflating)
@ -241,7 +239,7 @@ tr.skipreason td{{color:#8b949e;font-size:.78rem;font-style:italic;padding-top:0
<div class="hd">{FLOWER_SVG} <div class="hd">{FLOWER_SVG}
<div class="title"><h1>{recipe}</h1><span class="ver">{version}</span></div> <div class="title"><h1>{recipe}</h1><span class="ver">{version}</span></div>
<div class="lvl"><span class="num">{level}</span><span class="lbl">level</span></div></div> <div class="lvl"><span class="num">{level}</span><span class="lbl">level</span></div></div>
<div class="cap">{("<b>capped:</b> " + cap) if cap else "<b>full clean climb</b> — top level (6)"}</div> <div class="cap">{("<b>capped:</b> " + cap) if cap else "<b>full clean climb</b> — top level (4)"}</div>
<div class="body"><div class="tbl"><table>{rows}</table></div>{shot_html}</div> <div class="body"><div class="tbl"><table>{rows}</table></div>{shot_html}</div>
<div class="flags">{"".join(flag_bits)}</div> <div class="flags">{"".join(flag_bits)}</div>
</div></body></html>""" </div></body></html>"""

View File

@ -5,37 +5,39 @@ YunoHost semantics: **a gap caps the level** — you only earn level L if every
PASS. The first rung that is not a clean PASS (a real FAIL *or* genuinely N/A for this recipe) stops PASS. The first rung that is not a clean PASS (a real FAIL *or* genuinely N/A for this recipe) stops
the climb; `cap_reason` records why. This is deliberately conservative: presentation must NEVER make the climb; `cap_reason` records why. This is deliberately conservative: presentation must NEVER make
a run look greener than its tests (plan §6 cardinal guardrail), so an N/A rung caps just like a fail a run look greener than its tests (plan §6 cardinal guardrail), so an N/A rung caps just like a fail
(the L5 example in §4.1 — "recipes with no integration surface cap at L4 by definition" — is exactly — with a recorded reason so the level is *fair*, not inflated.
this: N/A caps, with a recorded reason so the level is *fair*, not inflated).
The ladder (§4.1): The ladder is the FOUR essential rungs every recipe is held to:
L0 — install failed / app never became healthy. L0 — install failed / app never became healthy.
L1 — Installs: deploys + passes health/readiness. L1 — Installs: deploys + passes health/readiness.
L2 — Upgrades: previous published version → PR version, stays healthy, data intact. L2 — Upgrades: previous published version → PR version, stays healthy, data intact.
L3 — Backup/restore: seeded data survives backup → wipe → restore. L3 — Backup/restore: seeded data survives backup → wipe → restore.
L4 — Functional: recipe-specific functional tests pass. L4 — Functional: recipe-specific functional tests pass.
L5 — Integration: SSO/OIDC + cross-app integration tests pass.
L6 — Recipe-local: the recipe repo's own tests/ (D4) pass and are merged. Integration (SSO/OIDC + cross-app) and recipe-local (the recipe repo's own tests/) are **OPTIONAL**
capabilities — they are NOT part of the level ladder and never cap it. They still run when present
(and SSO is still enforced for the run VERDICT via the deps/SSO checks in run_recipe_ci.py), but a
recipe without an SSO surface or without repo-local tests is simply not penalised on the level.
This module is PURE (no I/O) so it is cheaply unit-testable and the Adversary can re-run the unit This module is PURE (no I/O) so it is cheaply unit-testable and the Adversary can re-run the unit
test cold (`cc-ci-run -m pytest tests/unit/test_level.py -q`). The orchestrator test cold (`cc-ci-run -m pytest tests/unit/test_level.py -q`). The orchestrator
(`run_recipe_ci.py`) is responsible for translating its raw per-tier results + deps/SSO signals into (`run_recipe_ci.py`) is responsible for translating its raw per-tier results into the rung-status
the rung-status dict this function consumes; that mapping is documented in DECISIONS.md (Phase 3). dict this function consumes; that mapping is documented in DECISIONS.md (Phase 3).
Rung status vocabulary (each rung ∈ these three): Rung status vocabulary (each rung ∈ these three):
"pass" — the rung was exercised and passed. "pass" — the rung was exercised and passed.
"fail" — the rung was exercised and failed. "fail" — the rung was exercised and failed.
"na" — the rung does not apply to this recipe (e.g. only one published version → no upgrade; "na" — the rung does not apply to this recipe (e.g. only one published version → no upgrade;
not backup-capable; no SSO/integration surface; no recipe-local tests). N/A is NOT a not backup-capable). N/A is NOT a failure, but it DOES cap the climb (with a distinct
failure, but it DOES cap the climb (with a distinct cap_reason) so the level never cap_reason) so the level never overstates what was actually verified.
overstates what was actually verified.
""" """
from __future__ import annotations from __future__ import annotations
# The climbable rungs in ascending order. install (L1) is the foundation; L0 means install itself # The climbable rungs in ascending order. install (L1) is the foundation; L0 means install itself
# did not pass. Each later rung requires every earlier rung to be a clean PASS. # did not pass. Each later rung requires every earlier rung to be a clean PASS. These four are the
RUNGS = ("install", "upgrade", "backup_restore", "functional", "integration", "recipe_local") # ESSENTIAL rungs — integration/recipe-local are optional and deliberately NOT in this tuple.
RUNGS = ("install", "upgrade", "backup_restore", "functional")
# Human-readable label per rung level, for cap_reason + the summary card. # Human-readable label per rung level, for cap_reason + the summary card.
RUNG_LABEL = { RUNG_LABEL = {
@ -43,22 +45,20 @@ RUNG_LABEL = {
2: "upgrade (prev published → PR)", 2: "upgrade (prev published → PR)",
3: "backup/restore (data integrity)", 3: "backup/restore (data integrity)",
4: "functional (recipe-specific tests)", 4: "functional (recipe-specific tests)",
5: "integration (SSO/OIDC + cross-app)",
6: "recipe-local (recipe repo tests/)",
} }
VALID = {"pass", "fail", "na"} VALID = {"pass", "fail", "na"}
def compute_level(rungs: dict[str, str]) -> tuple[int, str]: def compute_level(rungs: dict[str, str]) -> tuple[int, str]:
"""Map a rung-status dict → (level 0..6, cap_reason). """Map a rung-status dict → (level 0..4, cap_reason).
`rungs` must contain a status in {"pass","fail","na"} for every name in RUNGS. The level is the `rungs` must contain a status in {"pass","fail","na"} for every name in RUNGS. The level is the
highest L such that rungs[1..L] are all "pass"; the first non-"pass" rung caps the climb. L0 is highest L such that rungs[1..L] are all "pass"; the first non-"pass" rung caps the climb. L0 is
returned when the install rung itself is not "pass" (install failed / never healthy). returned when the install rung itself is not "pass" (install failed / never healthy).
cap_reason explains where the climb stopped: cap_reason explains where the climb stopped:
- "" (empty) when the recipe earned the top rung (L6, full clean climb). - "" (empty) when the recipe earned the top rung (L4, full clean climb).
- "L<k> <label> FAILED" when a rung was exercised and failed. - "L<k> <label> FAILED" when a rung was exercised and failed.
- "L<k> <label> N/A" when a rung does not apply to this recipe. - "L<k> <label> N/A" when a rung does not apply to this recipe.
Returns the reason for the FIRST rung that stopped the climb (the binding constraint). Returns the reason for the FIRST rung that stopped the climb (the binding constraint).

View File

@ -134,41 +134,24 @@ def collect_stages(records: list[dict]) -> list[dict]:
return stages return stages
def _has_repo_local(records: list[dict]) -> bool:
return any(r.get("source") == "repo-local" for r in records)
def _repo_local_passed(records: list[dict]) -> bool:
repo = [r for r in records if r.get("source") == "repo-local"]
return bool(repo) and all(r.get("rc", 1) == 0 for r in repo)
def derive_rungs( def derive_rungs(
results: dict[str, str], results: dict[str, str],
*, *,
backup_capable: bool, backup_capable: bool,
declared: list[str] | None,
deps_ready: bool,
sso_unverified: bool,
has_custom: bool, has_custom: bool,
has_repo_local: bool,
repo_local_passed: bool,
) -> dict[str, str]: ) -> dict[str, str]:
"""Translate the orchestrator's tier results + deps/SSO signals into the rung-status dict """Translate the orchestrator's tier results into the rung-status dict harness.level consumes —
harness.level consumes. Documented in DECISIONS.md (Phase 3). Conservative by design — never the FOUR essential rungs only. Conservative by design — never reports a rung 'pass' it can't
reports a rung 'pass' it can't substantiate (cardinal guardrail: presentation never inflates). substantiate (cardinal guardrail: presentation never inflates).
L1 install : install tier pass. L1 install : install tier pass.
L2 upgrade : upgrade tier (skip → N/A: only one published version). L2 upgrade : upgrade tier (skip → N/A: only one published version).
L3 backup/res : backup AND restore tiers pass (N/A if not backup-capable). L3 backup/res : backup AND restore tiers pass (N/A if not backup-capable).
L4 functional : the recipe-specific functional (non-deps) tests pass — the custom tier, minus L4 functional : recipe-specific functional tests pass — the custom tier. N/A if none ran.
its SSO/integration tests. N/A if the recipe has no custom tests at all.
L5 integration: SSO/OIDC + cross-app. Applies ONLY if the recipe declares deps (else N/A — the Integration (SSO/OIDC) and recipe-local are OPTIONAL and intentionally NOT rungs here — they
"no integration surface caps at L4" rule, §4.1). pass iff deps wired never cap the level (SSO is still enforced for the run VERDICT in run_recipe_ci.py).
(deps_ready) and not sso_unverified and the custom tier didn't fail.
L6 recipe-loc : the recipe repo's own tests/ (repo-local source) ran and passed (N/A if none).
""" """
declared = declared or []
rungs: dict[str, str] = {} rungs: dict[str, str] = {}
rungs["install"] = level_mod.tier_to_rung(results.get("install")) rungs["install"] = level_mod.tier_to_rung(results.get("install"))
rungs["upgrade"] = level_mod.tier_to_rung(results.get("upgrade")) rungs["upgrade"] = level_mod.tier_to_rung(results.get("upgrade"))
@ -177,33 +160,12 @@ def derive_rungs(
) )
custom = results.get("custom") custom = results.get("custom")
# Functional rung (L4): the non-deps custom tests.
if not has_custom or custom == "skip" or custom is None: if not has_custom or custom == "skip" or custom is None:
rungs["functional"] = "na" rungs["functional"] = "na"
elif custom == "fail": elif custom == "fail":
# A custom test failed. With declared deps we cannot cheaply tell functional-vs-SSO apart, so
# conservatively fail the functional rung (caps at L3) — never inflate.
rungs["functional"] = "fail" rungs["functional"] = "fail"
else: # custom == "pass" else: # custom == "pass"
rungs["functional"] = "pass" rungs["functional"] = "pass"
# Integration rung (L5): only recipes with an SSO/integration surface (declared deps) can climb.
if not declared:
rungs["integration"] = "na"
elif sso_unverified or not deps_ready or custom == "fail":
# SSO not wired/verified, or a custom test failed → integration not verified.
rungs["integration"] = "fail"
elif custom == "pass":
rungs["integration"] = "pass"
else:
# declared deps but no custom tests ran — can't claim integration verified
rungs["integration"] = "na"
# Recipe-local rung (L6).
if not has_repo_local:
rungs["recipe_local"] = "na"
else:
rungs["recipe_local"] = "pass" if repo_local_passed else "fail"
return rungs return rungs
@ -235,9 +197,6 @@ def build_results(
records: list[dict], records: list[dict],
results: dict[str, str], results: dict[str, str],
backup_capable: bool, backup_capable: bool,
declared: list[str] | None,
deps_ready: bool,
sso_unverified: bool,
clean_teardown: bool, clean_teardown: bool,
no_secret_leak: bool, no_secret_leak: bool,
finished_ts: float | None, finished_ts: float | None,
@ -247,20 +206,11 @@ def build_results(
) -> dict: ) -> dict:
"""Assemble the full results.json dict (no I/O). `finished_ts` is passed in (the orchestrator """Assemble the full results.json dict (no I/O). `finished_ts` is passed in (the orchestrator
stamps it) so this stays pure and deterministic for unit tests. `expected_na` is the recipe's stamps it) so this stays pure and deterministic for unit tests. `expected_na` is the recipe's
declared intentional-N/A map (recipe_meta.EXPECTED_NA) used to distinguish a deliberate skip from declared intentional-skip map (recipe_meta.EXPECTED_NA) used to distinguish a deliberate skip from
accidentally-missing coverage.""" accidentally-missing coverage."""
stages = collect_stages(records) stages = collect_stages(records)
has_custom = any(r["tier"] == "custom" for r in records) has_custom = any(r["tier"] == "custom" for r in records)
rungs = derive_rungs( rungs = derive_rungs(results, backup_capable=backup_capable, has_custom=has_custom)
results,
backup_capable=backup_capable,
declared=declared,
deps_ready=deps_ready,
sso_unverified=sso_unverified,
has_custom=has_custom,
has_repo_local=_has_repo_local(records),
repo_local_passed=_repo_local_passed(records),
)
lvl, cap_reason = level_mod.compute_level(rungs) lvl, cap_reason = level_mod.compute_level(rungs)
# The rung that capped the climb (lowest non-pass), or None on a full climb — lets a consumer # The rung that capped the climb (lowest non-pass), or None on a full climb — lets a consumer
# (card/badge) tell whether the cap was an intentional skip, an unintentional one, or a failure. # (card/badge) tell whether the cap was an intentional skip, an unintentional one, or a failure.

View File

@ -1225,7 +1225,6 @@ def main() -> int:
# a failure here NEVER changes `overall` (R7 — cosmetics never block the pipeline). ---- # a failure here NEVER changes `overall` (R7 — cosmetics never block the pipeline). ----
data: dict | None = None data: dict | None = None
try: try:
sso_unverified = sso_dep_unverified(declared, deps_ready, requires_deps_skipped)
clean_teardown = (deploy_count == expected_deploy_count) and not dep_teardown_error clean_teardown = (deploy_count == expected_deploy_count) and not dep_teardown_error
data = results_mod.build_results( data = results_mod.build_results(
recipe=recipe, recipe=recipe,
@ -1235,14 +1234,11 @@ def main() -> int:
records=records, records=records,
results=results, results=results,
backup_capable=backup_cap, backup_capable=backup_cap,
declared=declared,
deps_ready=deps_ready,
sso_unverified=sso_unverified,
clean_teardown=clean_teardown, clean_teardown=clean_teardown,
no_secret_leak=True, # narrowed below by an actual scan of the serialised artifact no_secret_leak=True, # narrowed below by an actual scan of the serialised artifact
screenshot=screenshot_rel, # Phase 3 U1 (R4): relative PNG name iff capture succeeded screenshot=screenshot_rel, # Phase 3 U1 (R4): relative PNG name iff capture succeeded
finished_ts=time.time(), finished_ts=time.time(),
expected_na=meta.get("EXPECTED_NA"), # declared intentional-N/A map (recipe_meta) expected_na=meta.get("EXPECTED_NA"), # declared intentional-skip map (recipe_meta)
) )
# Real (if narrow) leak check: no known infra-secret value may appear in the artifact (R7). # Real (if narrow) leak check: no known infra-secret value may appear in the artifact (R7).
blob = json.dumps(data) blob = json.dumps(data)

View File

@ -4,14 +4,13 @@
DEPLOY_TIMEOUT = 120 DEPLOY_TIMEOUT = 120
HTTP_TIMEOUT = 90 HTTP_TIMEOUT = 90
# Rungs this recipe INTENTIONALLY skips, each with a reason. Any rung that is skipped (N/A) and is # Rungs this recipe INTENTIONALLY skips, each with a reason. Any essential rung skipped (N/A) and NOT
# NOT listed here is reported as an *unintentional* skip (a coverage gap to fill or declare). A skip # listed here is reported as an *unintentional* skip (a coverage gap to fill or declare). A skip still
# still caps the level either way — the harness never claims a rung it did not verify; this only # caps the level either way — the harness never claims a rung it did not verify; this only records
# records that the skip is deliberate. custom-html-tiny is a stateless static-web-server, so: # that the skip is deliberate. (The level ladder is the four essential rungs install/upgrade/
# backup_restore/functional; integration + recipe-local are optional and not leveled.)
# custom-html-tiny is a stateless static-web-server, so it has no backup surface:
EXPECTED_NA = { EXPECTED_NA = {
"backup_restore": "stateless static file server: serves an ephemeral content volume seeded at " "backup_restore": "stateless static file server: serves an ephemeral content volume seeded at "
"deploy, with no persistent/user data to back up or restore (no backupbot.backup label)", "deploy, with no persistent/user data to back up or restore (no backupbot.backup label)",
"integration": "no SSO/OIDC or cross-app surface — a static file server has no auth integration",
"recipe_local": "the upstream recipe ships no tests/ of its own; coverage is the cc-ci generic "
"install tier + the functional serve test",
} }

View File

@ -14,7 +14,7 @@ sys.path.insert(0, os.path.join(os.path.dirname(__file__), "..", "..", "runner")
from harness import card as C # noqa: E402 from harness import card as C # noqa: E402
def _data(level=4, cap="L5 integration (SSO/OIDC + cross-app) N/A"): def _data(level=3, cap="L4 functional (recipe-specific tests) N/A"):
return { return {
"recipe": "uptime-kuma", "recipe": "uptime-kuma",
"version": "1.23.0", "version": "1.23.0",

View File

@ -24,7 +24,7 @@ import dashboard # noqa: E402
def _row(**kw): def _row(**kw):
base = { base = {
"recipe": "custom-html", "status": "success", "number": 4, "ref": "db9a9502", "recipe": "custom-html", "status": "success", "number": 4, "ref": "db9a9502",
"version": "db9a95024e9d", "level": 4, "level_cap_reason": "L5 integration N/A", "version": "db9a95024e9d", "level": 4, "level_cap_reason": "",
"has_screenshot": True, "flags": {"clean_teardown": True, "no_secret_leak": True}, "has_screenshot": True, "flags": {"clean_teardown": True, "no_secret_leak": True},
"finished": 0, "url": "https://drone.x/cc-ci/4", "finished": 0, "url": "https://drone.x/cc-ci/4",
} }

View File

@ -19,33 +19,23 @@ def _rungs(
upgrade="pass", upgrade="pass",
backup_restore="pass", backup_restore="pass",
functional="pass", functional="pass",
integration="pass",
recipe_local="pass",
): ):
return { return {
"install": install, "install": install,
"upgrade": upgrade, "upgrade": upgrade,
"backup_restore": backup_restore, "backup_restore": backup_restore,
"functional": functional, "functional": functional,
"integration": integration,
"recipe_local": recipe_local,
} }
# ---- the U0 gate: L4-pass and L2-cap ---- # ---- the ladder: four essential rungs, top is L4 (functional) ----
def test_full_clean_climb_to_L6(): def test_full_clean_climb_to_L4():
# All four essential rungs pass → L4 (the top; integration/recipe-local are optional, not leveled).
lvl, reason = L.compute_level(_rungs()) lvl, reason = L.compute_level(_rungs())
assert lvl == 6
assert reason == ""
def test_climbs_through_L4_then_no_integration_surface_caps_at_L4():
# GATE: a recipe whose functional tests pass but has no SSO/integration surface caps at L4.
lvl, reason = L.compute_level(_rungs(integration="na", recipe_local="na"))
assert lvl == 4 assert lvl == 4
assert "L5" in reason and "N/A" in reason assert reason == ""
def test_fails_at_L2_capped_at_L1(): def test_fails_at_L2_capped_at_L1():
@ -69,34 +59,27 @@ def test_install_fail_is_L0():
def test_higher_pass_does_not_rescue_lower_na(): def test_higher_pass_does_not_rescue_lower_na():
# backup/restore N/A (stateless app) caps at L2 even though functional would pass. # backup/restore N/A (stateless app) caps at L2 even though functional would pass.
lvl, reason = L.compute_level(_rungs(backup_restore="na", functional="pass", integration="na")) lvl, reason = L.compute_level(_rungs(backup_restore="na", functional="pass"))
assert lvl == 2 assert lvl == 2
assert "L3" in reason and "N/A" in reason assert "L3" in reason and "N/A" in reason
def test_upgrade_na_caps_at_L1(): def test_upgrade_na_caps_at_L1():
# only one published version → no upgrade possible → N/A caps at L1. # only one published version → no upgrade possible → N/A caps at L1 (upgrade is essential).
lvl, reason = L.compute_level(_rungs(upgrade="na")) lvl, reason = L.compute_level(_rungs(upgrade="na"))
assert lvl == 1 assert lvl == 1
assert "L2" in reason and "N/A" in reason assert "L2" in reason and "N/A" in reason
def test_integration_fail_caps_at_L4(): def test_functional_na_caps_at_L3():
# SSO declared but unverified (failed) → integration rung fails → cap at L4. # no recipe-specific functional tests → functional N/A caps at L3.
lvl, reason = L.compute_level(_rungs(integration="fail", recipe_local="na")) lvl, reason = L.compute_level(_rungs(functional="na"))
assert lvl == 4 assert lvl == 3
assert "L5" in reason and "FAILED" in reason assert "L4" in reason and "N/A" in reason
def test_recipe_local_na_caps_at_L5():
# SSO passes but no recipe-local tests → cap at L5 (L6 N/A).
lvl, reason = L.compute_level(_rungs(recipe_local="na"))
assert lvl == 5
assert "L6" in reason and "N/A" in reason
def test_functional_fail_caps_at_L3(): def test_functional_fail_caps_at_L3():
lvl, reason = L.compute_level(_rungs(functional="fail", integration="na")) lvl, reason = L.compute_level(_rungs(functional="fail"))
assert lvl == 3 assert lvl == 3
assert "L4" in reason and "FAILED" in reason assert "L4" in reason and "FAILED" in reason

View File

@ -105,83 +105,31 @@ def _results(**kw):
return base return base
def test_derive_rungs_full_stateful_sso(): def test_derive_rungs_full_climb_four_essential():
rungs = R.derive_rungs( rungs = R.derive_rungs(_results(), backup_capable=True, has_custom=True)
_results(), # only the four essential rungs — integration/recipe-local are optional, not produced here.
backup_capable=True,
declared=["keycloak"],
deps_ready=True,
sso_unverified=False,
has_custom=True,
has_repo_local=False,
repo_local_passed=False,
)
assert rungs == { assert rungs == {
"install": "pass", "install": "pass",
"upgrade": "pass", "upgrade": "pass",
"backup_restore": "pass", "backup_restore": "pass",
"functional": "pass", "functional": "pass",
"integration": "pass",
"recipe_local": "na",
} }
def test_derive_rungs_no_sso_surface_is_integration_na(): def test_derive_rungs_stateless_backup_and_functional_na():
rungs = R.derive_rungs(
_results(),
backup_capable=True,
declared=[],
deps_ready=True,
sso_unverified=False,
has_custom=True,
has_repo_local=False,
repo_local_passed=False,
)
assert rungs["integration"] == "na"
assert rungs["functional"] == "pass"
def test_derive_rungs_stateless_backup_na():
rungs = R.derive_rungs( rungs = R.derive_rungs(
_results(backup="skip", restore="skip", custom="skip"), _results(backup="skip", restore="skip", custom="skip"),
backup_capable=False, backup_capable=False,
declared=[],
deps_ready=True,
sso_unverified=False,
has_custom=False, has_custom=False,
has_repo_local=False,
repo_local_passed=False,
) )
assert rungs["backup_restore"] == "na" assert rungs["backup_restore"] == "na"
assert rungs["functional"] == "na" assert rungs["functional"] == "na"
assert "integration" not in rungs and "recipe_local" not in rungs
def test_derive_rungs_sso_unverified_is_integration_fail(): def test_derive_rungs_functional_fail():
rungs = R.derive_rungs( rungs = R.derive_rungs(_results(custom="fail"), backup_capable=True, has_custom=True)
_results(), assert rungs["functional"] == "fail"
backup_capable=True,
declared=["keycloak"],
deps_ready=False,
sso_unverified=True,
has_custom=True,
has_repo_local=False,
repo_local_passed=False,
)
assert rungs["integration"] == "fail"
def test_derive_rungs_repo_local_pass():
rungs = R.derive_rungs(
_results(),
backup_capable=True,
declared=[],
deps_ready=True,
sso_unverified=False,
has_custom=True,
has_repo_local=True,
repo_local_passed=True,
)
assert rungs["recipe_local"] == "pass"
# ---- build_results: end-to-end incl level + flags ---- # ---- build_results: end-to-end incl level + flags ----
@ -212,16 +160,13 @@ def test_build_results_level_and_flags(tmp_path):
records=recs, records=recs,
results=_results(), results=_results(),
backup_capable=True, backup_capable=True,
declared=[],
deps_ready=True,
sso_unverified=False,
clean_teardown=True, clean_teardown=True,
no_secret_leak=True, no_secret_leak=True,
finished_ts=1234.0, finished_ts=1234.0,
) )
# stateful, functional pass, no SSO surface, no repo-local → caps at L4 # all four essential rungs pass → full climb to L4 (the top), no cap
assert data["level"] == 4 assert data["level"] == 4
assert "L5" in data["level_cap_reason"] assert data["level_cap_reason"] == ""
assert data["recipe"] == "hedgedoc" assert data["recipe"] == "hedgedoc"
assert data["ref"] == "deadbeefcafe" assert data["ref"] == "deadbeefcafe"
assert data["flags"] == {"clean_teardown": True, "no_secret_leak": True} assert data["flags"] == {"clean_teardown": True, "no_secret_leak": True}
@ -246,9 +191,6 @@ def test_build_results_capped_at_L1_on_upgrade_fail(tmp_path):
records=recs, records=recs,
results=_results(upgrade="fail"), results=_results(upgrade="fail"),
backup_capable=True, backup_capable=True,
declared=[],
deps_ready=True,
sso_unverified=False,
clean_teardown=True, clean_teardown=True,
no_secret_leak=True, no_secret_leak=True,
finished_ts=0.0, finished_ts=0.0,
@ -266,8 +208,6 @@ def _rungs(**kw):
"upgrade": "pass", "upgrade": "pass",
"backup_restore": "pass", "backup_restore": "pass",
"functional": "pass", "functional": "pass",
"integration": "na",
"recipe_local": "na",
} }
base.update(kw) base.update(kw)
return base return base
@ -276,16 +216,16 @@ def _rungs(**kw):
def test_skips_intentional_vs_unintentional(): def test_skips_intentional_vs_unintentional():
rungs = _rungs(backup_restore="na", functional="na") rungs = _rungs(backup_restore="na", functional="na")
sk = R.skips(rungs, {"backup_restore": "stateless static server"}) sk = R.skips(rungs, {"backup_restore": "stateless static server"})
# backup_restore is declared (intentional, with reason); everything else skipped is unintentional. # backup_restore is declared (intentional, with reason); functional skipped but not declared.
assert sk["intentional"] == {"backup_restore": "stateless static server"} assert sk["intentional"] == {"backup_restore": "stateless static server"}
assert sk["unintentional"] == ["functional", "integration", "recipe_local"] assert sk["unintentional"] == ["functional"]
def test_skips_none_declared_all_unintentional(): def test_skips_none_declared_all_unintentional():
rungs = _rungs(backup_restore="na") rungs = _rungs(backup_restore="na")
sk = R.skips(rungs, None) sk = R.skips(rungs, None)
assert sk["intentional"] == {} assert sk["intentional"] == {}
assert sk["unintentional"] == ["backup_restore", "integration", "recipe_local"] assert sk["unintentional"] == ["backup_restore"]
def test_skips_declaration_only_counts_when_actually_skipped(): def test_skips_declaration_only_counts_when_actually_skipped():
@ -323,17 +263,10 @@ def test_build_results_threads_expected_na(tmp_path):
records=recs, records=recs,
results=_results(backup="skip", restore="skip"), # custom=pass (default) → functional pass results=_results(backup="skip", restore="skip"), # custom=pass (default) → functional pass
backup_capable=False, # no backupbot label → backup_restore skipped (N/A) backup_capable=False, # no backupbot label → backup_restore skipped (N/A)
declared=[],
deps_ready=True,
sso_unverified=False,
clean_teardown=True, clean_teardown=True,
no_secret_leak=True, no_secret_leak=True,
finished_ts=0.0, finished_ts=0.0,
expected_na={ expected_na={"backup_restore": "stateless static file server"},
"backup_restore": "stateless static file server",
"integration": "no SSO surface",
"recipe_local": "no upstream tests/",
},
) )
# backup_restore skip still caps at L2 (never inflates) — even though functional passes above it, # backup_restore skip still caps at L2 (never inflates) — even though functional passes above it,
# the skip caps the climb — but it's the declared (intentional) rung that capped. # the skip caps the climb — but it's the declared (intentional) rung that capped.
@ -342,7 +275,7 @@ def test_build_results_threads_expected_na(tmp_path):
assert data["level_cap_rung"] == "backup_restore" assert data["level_cap_rung"] == "backup_restore"
assert data["rungs"]["functional"] == "pass" assert data["rungs"]["functional"] == "pass"
assert data["skips"]["intentional"]["backup_restore"] == "stateless static file server" assert data["skips"]["intentional"]["backup_restore"] == "stateless static file server"
assert data["skips"]["unintentional"] == [] # every skip accounted for → fully clean assert data["skips"]["unintentional"] == [] # backup_restore declared; functional passed → clean
def test_write_results_roundtrip(tmp_path): def test_write_results_roundtrip(tmp_path):