refactor(level): four essential rungs only — integration & recipe-local are optional
Some checks failed
continuous-integration/drone/push Build is failing
Some checks failed
continuous-integration/drone/push Build is failing
Per operator: the level ladder is now the FOUR essential rungs every recipe is held to — install, upgrade (essential), backup/restore, functional (top = L4). Integration (SSO/OIDC) and recipe-local are OPTIONAL capabilities: they no longer appear as level rungs or skip rows and never cap the level. SSO is still enforced for the run VERDICT (unchanged in run_recipe_ci.py); it just doesn't affect the level. derive_rungs simplified accordingly (drops declared/deps/sso/repo-local inputs). custom-html-tiny's EXPECTED_NA is back to just backup_restore. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
autonomic-bot
@ -141,14 +141,12 @@ def _stage_rows(stages: list[dict]) -> str:
|
||||
return "\n".join(rows) or '<tr><td colspan="3">no stages</td></tr>'
|
||||
|
||||
|
||||
# Friendly rung labels for the skip rows.
|
||||
# Friendly rung labels for the skip rows (the four essential rungs).
|
||||
RUNG_LABEL = {
|
||||
"install": "install",
|
||||
"upgrade": "upgrade",
|
||||
"backup_restore": "backup/restore",
|
||||
"functional": "functional",
|
||||
"integration": "integration",
|
||||
"recipe_local": "recipe-local",
|
||||
}
|
||||
SKIP_GREEN = "#57ab5a" # muted green — an intentional skip reads like a pass (but labelled, never inflating)
|
||||
|
||||
@ -241,7 +239,7 @@ tr.skipreason td{{color:#8b949e;font-size:.78rem;font-style:italic;padding-top:0
|
||||
<div class="hd">{FLOWER_SVG}
|
||||
<div class="title"><h1>{recipe}</h1><span class="ver">{version}</span></div>
|
||||
<div class="lvl"><span class="num">{level}</span><span class="lbl">level</span></div></div>
|
||||
<div class="cap">{("<b>capped:</b> " + cap) if cap else "<b>full clean climb</b> — top level (6)"}</div>
|
||||
<div class="cap">{("<b>capped:</b> " + cap) if cap else "<b>full clean climb</b> — top level (4)"}</div>
|
||||
<div class="body"><div class="tbl"><table>{rows}</table></div>{shot_html}</div>
|
||||
<div class="flags">{"".join(flag_bits)}</div>
|
||||
</div></body></html>"""
|
||||
|
||||
@ -5,37 +5,39 @@ YunoHost semantics: **a gap caps the level** — you only earn level L if every
|
||||
PASS. The first rung that is not a clean PASS (a real FAIL *or* genuinely N/A for this recipe) stops
|
||||
the climb; `cap_reason` records why. This is deliberately conservative: presentation must NEVER make
|
||||
a run look greener than its tests (plan §6 cardinal guardrail), so an N/A rung caps just like a fail
|
||||
(the L5 example in §4.1 — "recipes with no integration surface cap at L4 by definition" — is exactly
|
||||
this: N/A caps, with a recorded reason so the level is *fair*, not inflated).
|
||||
— with a recorded reason so the level is *fair*, not inflated.
|
||||
|
||||
The ladder (§4.1):
|
||||
The ladder is the FOUR essential rungs every recipe is held to:
|
||||
L0 — install failed / app never became healthy.
|
||||
L1 — Installs: deploys + passes health/readiness.
|
||||
L2 — Upgrades: previous published version → PR version, stays healthy, data intact.
|
||||
L3 — Backup/restore: seeded data survives backup → wipe → restore.
|
||||
L4 — Functional: recipe-specific functional tests pass.
|
||||
L5 — Integration: SSO/OIDC + cross-app integration tests pass.
|
||||
L6 — Recipe-local: the recipe repo's own tests/ (D4) pass and are merged.
|
||||
|
||||
Integration (SSO/OIDC + cross-app) and recipe-local (the recipe repo's own tests/) are **OPTIONAL**
|
||||
capabilities — they are NOT part of the level ladder and never cap it. They still run when present
|
||||
(and SSO is still enforced for the run VERDICT via the deps/SSO checks in run_recipe_ci.py), but a
|
||||
recipe without an SSO surface or without repo-local tests is simply not penalised on the level.
|
||||
|
||||
This module is PURE (no I/O) so it is cheaply unit-testable and the Adversary can re-run the unit
|
||||
test cold (`cc-ci-run -m pytest tests/unit/test_level.py -q`). The orchestrator
|
||||
(`run_recipe_ci.py`) is responsible for translating its raw per-tier results + deps/SSO signals into
|
||||
the rung-status dict this function consumes; that mapping is documented in DECISIONS.md (Phase 3).
|
||||
(`run_recipe_ci.py`) is responsible for translating its raw per-tier results into the rung-status
|
||||
dict this function consumes; that mapping is documented in DECISIONS.md (Phase 3).
|
||||
|
||||
Rung status vocabulary (each rung ∈ these three):
|
||||
"pass" — the rung was exercised and passed.
|
||||
"fail" — the rung was exercised and failed.
|
||||
"na" — the rung does not apply to this recipe (e.g. only one published version → no upgrade;
|
||||
not backup-capable; no SSO/integration surface; no recipe-local tests). N/A is NOT a
|
||||
failure, but it DOES cap the climb (with a distinct cap_reason) so the level never
|
||||
overstates what was actually verified.
|
||||
not backup-capable). N/A is NOT a failure, but it DOES cap the climb (with a distinct
|
||||
cap_reason) so the level never overstates what was actually verified.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
# The climbable rungs in ascending order. install (L1) is the foundation; L0 means install itself
|
||||
# did not pass. Each later rung requires every earlier rung to be a clean PASS.
|
||||
RUNGS = ("install", "upgrade", "backup_restore", "functional", "integration", "recipe_local")
|
||||
# did not pass. Each later rung requires every earlier rung to be a clean PASS. These four are the
|
||||
# ESSENTIAL rungs — integration/recipe-local are optional and deliberately NOT in this tuple.
|
||||
RUNGS = ("install", "upgrade", "backup_restore", "functional")
|
||||
|
||||
# Human-readable label per rung level, for cap_reason + the summary card.
|
||||
RUNG_LABEL = {
|
||||
@ -43,22 +45,20 @@ RUNG_LABEL = {
|
||||
2: "upgrade (prev published → PR)",
|
||||
3: "backup/restore (data integrity)",
|
||||
4: "functional (recipe-specific tests)",
|
||||
5: "integration (SSO/OIDC + cross-app)",
|
||||
6: "recipe-local (recipe repo tests/)",
|
||||
}
|
||||
|
||||
VALID = {"pass", "fail", "na"}
|
||||
|
||||
|
||||
def compute_level(rungs: dict[str, str]) -> tuple[int, str]:
|
||||
"""Map a rung-status dict → (level 0..6, cap_reason).
|
||||
"""Map a rung-status dict → (level 0..4, cap_reason).
|
||||
|
||||
`rungs` must contain a status in {"pass","fail","na"} for every name in RUNGS. The level is the
|
||||
highest L such that rungs[1..L] are all "pass"; the first non-"pass" rung caps the climb. L0 is
|
||||
returned when the install rung itself is not "pass" (install failed / never healthy).
|
||||
|
||||
cap_reason explains where the climb stopped:
|
||||
- "" (empty) when the recipe earned the top rung (L6, full clean climb).
|
||||
- "" (empty) when the recipe earned the top rung (L4, full clean climb).
|
||||
- "L<k> <label> FAILED" when a rung was exercised and failed.
|
||||
- "L<k> <label> N/A" when a rung does not apply to this recipe.
|
||||
Returns the reason for the FIRST rung that stopped the climb (the binding constraint).
|
||||
|
||||
@ -134,41 +134,24 @@ def collect_stages(records: list[dict]) -> list[dict]:
|
||||
return stages
|
||||
|
||||
|
||||
def _has_repo_local(records: list[dict]) -> bool:
|
||||
return any(r.get("source") == "repo-local" for r in records)
|
||||
|
||||
|
||||
def _repo_local_passed(records: list[dict]) -> bool:
|
||||
repo = [r for r in records if r.get("source") == "repo-local"]
|
||||
return bool(repo) and all(r.get("rc", 1) == 0 for r in repo)
|
||||
|
||||
|
||||
def derive_rungs(
|
||||
results: dict[str, str],
|
||||
*,
|
||||
backup_capable: bool,
|
||||
declared: list[str] | None,
|
||||
deps_ready: bool,
|
||||
sso_unverified: bool,
|
||||
has_custom: bool,
|
||||
has_repo_local: bool,
|
||||
repo_local_passed: bool,
|
||||
) -> dict[str, str]:
|
||||
"""Translate the orchestrator's tier results + deps/SSO signals into the rung-status dict
|
||||
harness.level consumes. Documented in DECISIONS.md (Phase 3). Conservative by design — never
|
||||
reports a rung 'pass' it can't substantiate (cardinal guardrail: presentation never inflates).
|
||||
"""Translate the orchestrator's tier results into the rung-status dict harness.level consumes —
|
||||
the FOUR essential rungs only. Conservative by design — never reports a rung 'pass' it can't
|
||||
substantiate (cardinal guardrail: presentation never inflates).
|
||||
|
||||
L1 install : install tier pass.
|
||||
L2 upgrade : upgrade tier (skip → N/A: only one published version).
|
||||
L3 backup/res : backup AND restore tiers pass (N/A if not backup-capable).
|
||||
L4 functional : the recipe-specific functional (non-deps) tests pass — the custom tier, minus
|
||||
its SSO/integration tests. N/A if the recipe has no custom tests at all.
|
||||
L5 integration: SSO/OIDC + cross-app. Applies ONLY if the recipe declares deps (else N/A — the
|
||||
"no integration surface caps at L4" rule, §4.1). pass iff deps wired
|
||||
(deps_ready) and not sso_unverified and the custom tier didn't fail.
|
||||
L6 recipe-loc : the recipe repo's own tests/ (repo-local source) ran and passed (N/A if none).
|
||||
L4 functional : recipe-specific functional tests pass — the custom tier. N/A if none ran.
|
||||
|
||||
Integration (SSO/OIDC) and recipe-local are OPTIONAL and intentionally NOT rungs here — they
|
||||
never cap the level (SSO is still enforced for the run VERDICT in run_recipe_ci.py).
|
||||
"""
|
||||
declared = declared or []
|
||||
rungs: dict[str, str] = {}
|
||||
rungs["install"] = level_mod.tier_to_rung(results.get("install"))
|
||||
rungs["upgrade"] = level_mod.tier_to_rung(results.get("upgrade"))
|
||||
@ -177,33 +160,12 @@ def derive_rungs(
|
||||
)
|
||||
|
||||
custom = results.get("custom")
|
||||
# Functional rung (L4): the non-deps custom tests.
|
||||
if not has_custom or custom == "skip" or custom is None:
|
||||
rungs["functional"] = "na"
|
||||
elif custom == "fail":
|
||||
# A custom test failed. With declared deps we cannot cheaply tell functional-vs-SSO apart, so
|
||||
# conservatively fail the functional rung (caps at L3) — never inflate.
|
||||
rungs["functional"] = "fail"
|
||||
else: # custom == "pass"
|
||||
rungs["functional"] = "pass"
|
||||
|
||||
# Integration rung (L5): only recipes with an SSO/integration surface (declared deps) can climb.
|
||||
if not declared:
|
||||
rungs["integration"] = "na"
|
||||
elif sso_unverified or not deps_ready or custom == "fail":
|
||||
# SSO not wired/verified, or a custom test failed → integration not verified.
|
||||
rungs["integration"] = "fail"
|
||||
elif custom == "pass":
|
||||
rungs["integration"] = "pass"
|
||||
else:
|
||||
# declared deps but no custom tests ran — can't claim integration verified
|
||||
rungs["integration"] = "na"
|
||||
|
||||
# Recipe-local rung (L6).
|
||||
if not has_repo_local:
|
||||
rungs["recipe_local"] = "na"
|
||||
else:
|
||||
rungs["recipe_local"] = "pass" if repo_local_passed else "fail"
|
||||
return rungs
|
||||
|
||||
|
||||
@ -235,9 +197,6 @@ def build_results(
|
||||
records: list[dict],
|
||||
results: dict[str, str],
|
||||
backup_capable: bool,
|
||||
declared: list[str] | None,
|
||||
deps_ready: bool,
|
||||
sso_unverified: bool,
|
||||
clean_teardown: bool,
|
||||
no_secret_leak: bool,
|
||||
finished_ts: float | None,
|
||||
@ -247,20 +206,11 @@ def build_results(
|
||||
) -> dict:
|
||||
"""Assemble the full results.json dict (no I/O). `finished_ts` is passed in (the orchestrator
|
||||
stamps it) so this stays pure and deterministic for unit tests. `expected_na` is the recipe's
|
||||
declared intentional-N/A map (recipe_meta.EXPECTED_NA) used to distinguish a deliberate skip from
|
||||
declared intentional-skip map (recipe_meta.EXPECTED_NA) used to distinguish a deliberate skip from
|
||||
accidentally-missing coverage."""
|
||||
stages = collect_stages(records)
|
||||
has_custom = any(r["tier"] == "custom" for r in records)
|
||||
rungs = derive_rungs(
|
||||
results,
|
||||
backup_capable=backup_capable,
|
||||
declared=declared,
|
||||
deps_ready=deps_ready,
|
||||
sso_unverified=sso_unverified,
|
||||
has_custom=has_custom,
|
||||
has_repo_local=_has_repo_local(records),
|
||||
repo_local_passed=_repo_local_passed(records),
|
||||
)
|
||||
rungs = derive_rungs(results, backup_capable=backup_capable, has_custom=has_custom)
|
||||
lvl, cap_reason = level_mod.compute_level(rungs)
|
||||
# The rung that capped the climb (lowest non-pass), or None on a full climb — lets a consumer
|
||||
# (card/badge) tell whether the cap was an intentional skip, an unintentional one, or a failure.
|
||||
|
||||
@ -1225,7 +1225,6 @@ def main() -> int:
|
||||
# a failure here NEVER changes `overall` (R7 — cosmetics never block the pipeline). ----
|
||||
data: dict | None = None
|
||||
try:
|
||||
sso_unverified = sso_dep_unverified(declared, deps_ready, requires_deps_skipped)
|
||||
clean_teardown = (deploy_count == expected_deploy_count) and not dep_teardown_error
|
||||
data = results_mod.build_results(
|
||||
recipe=recipe,
|
||||
@ -1235,14 +1234,11 @@ def main() -> int:
|
||||
records=records,
|
||||
results=results,
|
||||
backup_capable=backup_cap,
|
||||
declared=declared,
|
||||
deps_ready=deps_ready,
|
||||
sso_unverified=sso_unverified,
|
||||
clean_teardown=clean_teardown,
|
||||
no_secret_leak=True, # narrowed below by an actual scan of the serialised artifact
|
||||
screenshot=screenshot_rel, # Phase 3 U1 (R4): relative PNG name iff capture succeeded
|
||||
finished_ts=time.time(),
|
||||
expected_na=meta.get("EXPECTED_NA"), # declared intentional-N/A map (recipe_meta)
|
||||
expected_na=meta.get("EXPECTED_NA"), # declared intentional-skip map (recipe_meta)
|
||||
)
|
||||
# Real (if narrow) leak check: no known infra-secret value may appear in the artifact (R7).
|
||||
blob = json.dumps(data)
|
||||
|
||||
@ -4,14 +4,13 @@
|
||||
DEPLOY_TIMEOUT = 120
|
||||
HTTP_TIMEOUT = 90
|
||||
|
||||
# Rungs this recipe INTENTIONALLY skips, each with a reason. Any rung that is skipped (N/A) and is
|
||||
# NOT listed here is reported as an *unintentional* skip (a coverage gap to fill or declare). A skip
|
||||
# still caps the level either way — the harness never claims a rung it did not verify; this only
|
||||
# records that the skip is deliberate. custom-html-tiny is a stateless static-web-server, so:
|
||||
# Rungs this recipe INTENTIONALLY skips, each with a reason. Any essential rung skipped (N/A) and NOT
|
||||
# listed here is reported as an *unintentional* skip (a coverage gap to fill or declare). A skip still
|
||||
# caps the level either way — the harness never claims a rung it did not verify; this only records
|
||||
# that the skip is deliberate. (The level ladder is the four essential rungs install/upgrade/
|
||||
# backup_restore/functional; integration + recipe-local are optional and not leveled.)
|
||||
# custom-html-tiny is a stateless static-web-server, so it has no backup surface:
|
||||
EXPECTED_NA = {
|
||||
"backup_restore": "stateless static file server: serves an ephemeral content volume seeded at "
|
||||
"deploy, with no persistent/user data to back up or restore (no backupbot.backup label)",
|
||||
"integration": "no SSO/OIDC or cross-app surface — a static file server has no auth integration",
|
||||
"recipe_local": "the upstream recipe ships no tests/ of its own; coverage is the cc-ci generic "
|
||||
"install tier + the functional serve test",
|
||||
}
|
||||
|
||||
@ -14,7 +14,7 @@ sys.path.insert(0, os.path.join(os.path.dirname(__file__), "..", "..", "runner")
|
||||
from harness import card as C # noqa: E402
|
||||
|
||||
|
||||
def _data(level=4, cap="L5 integration (SSO/OIDC + cross-app) N/A"):
|
||||
def _data(level=3, cap="L4 functional (recipe-specific tests) N/A"):
|
||||
return {
|
||||
"recipe": "uptime-kuma",
|
||||
"version": "1.23.0",
|
||||
|
||||
@ -24,7 +24,7 @@ import dashboard # noqa: E402
|
||||
def _row(**kw):
|
||||
base = {
|
||||
"recipe": "custom-html", "status": "success", "number": 4, "ref": "db9a9502",
|
||||
"version": "db9a95024e9d", "level": 4, "level_cap_reason": "L5 integration N/A",
|
||||
"version": "db9a95024e9d", "level": 4, "level_cap_reason": "",
|
||||
"has_screenshot": True, "flags": {"clean_teardown": True, "no_secret_leak": True},
|
||||
"finished": 0, "url": "https://drone.x/cc-ci/4",
|
||||
}
|
||||
|
||||
@ -19,33 +19,23 @@ def _rungs(
|
||||
upgrade="pass",
|
||||
backup_restore="pass",
|
||||
functional="pass",
|
||||
integration="pass",
|
||||
recipe_local="pass",
|
||||
):
|
||||
return {
|
||||
"install": install,
|
||||
"upgrade": upgrade,
|
||||
"backup_restore": backup_restore,
|
||||
"functional": functional,
|
||||
"integration": integration,
|
||||
"recipe_local": recipe_local,
|
||||
}
|
||||
|
||||
|
||||
# ---- the U0 gate: L4-pass and L2-cap ----
|
||||
# ---- the ladder: four essential rungs, top is L4 (functional) ----
|
||||
|
||||
|
||||
def test_full_clean_climb_to_L6():
|
||||
def test_full_clean_climb_to_L4():
|
||||
# All four essential rungs pass → L4 (the top; integration/recipe-local are optional, not leveled).
|
||||
lvl, reason = L.compute_level(_rungs())
|
||||
assert lvl == 6
|
||||
assert reason == ""
|
||||
|
||||
|
||||
def test_climbs_through_L4_then_no_integration_surface_caps_at_L4():
|
||||
# GATE: a recipe whose functional tests pass but has no SSO/integration surface caps at L4.
|
||||
lvl, reason = L.compute_level(_rungs(integration="na", recipe_local="na"))
|
||||
assert lvl == 4
|
||||
assert "L5" in reason and "N/A" in reason
|
||||
assert reason == ""
|
||||
|
||||
|
||||
def test_fails_at_L2_capped_at_L1():
|
||||
@ -69,34 +59,27 @@ def test_install_fail_is_L0():
|
||||
|
||||
def test_higher_pass_does_not_rescue_lower_na():
|
||||
# backup/restore N/A (stateless app) caps at L2 even though functional would pass.
|
||||
lvl, reason = L.compute_level(_rungs(backup_restore="na", functional="pass", integration="na"))
|
||||
lvl, reason = L.compute_level(_rungs(backup_restore="na", functional="pass"))
|
||||
assert lvl == 2
|
||||
assert "L3" in reason and "N/A" in reason
|
||||
|
||||
|
||||
def test_upgrade_na_caps_at_L1():
|
||||
# only one published version → no upgrade possible → N/A caps at L1.
|
||||
# only one published version → no upgrade possible → N/A caps at L1 (upgrade is essential).
|
||||
lvl, reason = L.compute_level(_rungs(upgrade="na"))
|
||||
assert lvl == 1
|
||||
assert "L2" in reason and "N/A" in reason
|
||||
|
||||
|
||||
def test_integration_fail_caps_at_L4():
|
||||
# SSO declared but unverified (failed) → integration rung fails → cap at L4.
|
||||
lvl, reason = L.compute_level(_rungs(integration="fail", recipe_local="na"))
|
||||
assert lvl == 4
|
||||
assert "L5" in reason and "FAILED" in reason
|
||||
|
||||
|
||||
def test_recipe_local_na_caps_at_L5():
|
||||
# SSO passes but no recipe-local tests → cap at L5 (L6 N/A).
|
||||
lvl, reason = L.compute_level(_rungs(recipe_local="na"))
|
||||
assert lvl == 5
|
||||
assert "L6" in reason and "N/A" in reason
|
||||
def test_functional_na_caps_at_L3():
|
||||
# no recipe-specific functional tests → functional N/A caps at L3.
|
||||
lvl, reason = L.compute_level(_rungs(functional="na"))
|
||||
assert lvl == 3
|
||||
assert "L4" in reason and "N/A" in reason
|
||||
|
||||
|
||||
def test_functional_fail_caps_at_L3():
|
||||
lvl, reason = L.compute_level(_rungs(functional="fail", integration="na"))
|
||||
lvl, reason = L.compute_level(_rungs(functional="fail"))
|
||||
assert lvl == 3
|
||||
assert "L4" in reason and "FAILED" in reason
|
||||
|
||||
|
||||
@ -105,83 +105,31 @@ def _results(**kw):
|
||||
return base
|
||||
|
||||
|
||||
def test_derive_rungs_full_stateful_sso():
|
||||
rungs = R.derive_rungs(
|
||||
_results(),
|
||||
backup_capable=True,
|
||||
declared=["keycloak"],
|
||||
deps_ready=True,
|
||||
sso_unverified=False,
|
||||
has_custom=True,
|
||||
has_repo_local=False,
|
||||
repo_local_passed=False,
|
||||
)
|
||||
def test_derive_rungs_full_climb_four_essential():
|
||||
rungs = R.derive_rungs(_results(), backup_capable=True, has_custom=True)
|
||||
# only the four essential rungs — integration/recipe-local are optional, not produced here.
|
||||
assert rungs == {
|
||||
"install": "pass",
|
||||
"upgrade": "pass",
|
||||
"backup_restore": "pass",
|
||||
"functional": "pass",
|
||||
"integration": "pass",
|
||||
"recipe_local": "na",
|
||||
}
|
||||
|
||||
|
||||
def test_derive_rungs_no_sso_surface_is_integration_na():
|
||||
rungs = R.derive_rungs(
|
||||
_results(),
|
||||
backup_capable=True,
|
||||
declared=[],
|
||||
deps_ready=True,
|
||||
sso_unverified=False,
|
||||
has_custom=True,
|
||||
has_repo_local=False,
|
||||
repo_local_passed=False,
|
||||
)
|
||||
assert rungs["integration"] == "na"
|
||||
assert rungs["functional"] == "pass"
|
||||
|
||||
|
||||
def test_derive_rungs_stateless_backup_na():
|
||||
def test_derive_rungs_stateless_backup_and_functional_na():
|
||||
rungs = R.derive_rungs(
|
||||
_results(backup="skip", restore="skip", custom="skip"),
|
||||
backup_capable=False,
|
||||
declared=[],
|
||||
deps_ready=True,
|
||||
sso_unverified=False,
|
||||
has_custom=False,
|
||||
has_repo_local=False,
|
||||
repo_local_passed=False,
|
||||
)
|
||||
assert rungs["backup_restore"] == "na"
|
||||
assert rungs["functional"] == "na"
|
||||
assert "integration" not in rungs and "recipe_local" not in rungs
|
||||
|
||||
|
||||
def test_derive_rungs_sso_unverified_is_integration_fail():
|
||||
rungs = R.derive_rungs(
|
||||
_results(),
|
||||
backup_capable=True,
|
||||
declared=["keycloak"],
|
||||
deps_ready=False,
|
||||
sso_unverified=True,
|
||||
has_custom=True,
|
||||
has_repo_local=False,
|
||||
repo_local_passed=False,
|
||||
)
|
||||
assert rungs["integration"] == "fail"
|
||||
|
||||
|
||||
def test_derive_rungs_repo_local_pass():
|
||||
rungs = R.derive_rungs(
|
||||
_results(),
|
||||
backup_capable=True,
|
||||
declared=[],
|
||||
deps_ready=True,
|
||||
sso_unverified=False,
|
||||
has_custom=True,
|
||||
has_repo_local=True,
|
||||
repo_local_passed=True,
|
||||
)
|
||||
assert rungs["recipe_local"] == "pass"
|
||||
def test_derive_rungs_functional_fail():
|
||||
rungs = R.derive_rungs(_results(custom="fail"), backup_capable=True, has_custom=True)
|
||||
assert rungs["functional"] == "fail"
|
||||
|
||||
|
||||
# ---- build_results: end-to-end incl level + flags ----
|
||||
@ -212,16 +160,13 @@ def test_build_results_level_and_flags(tmp_path):
|
||||
records=recs,
|
||||
results=_results(),
|
||||
backup_capable=True,
|
||||
declared=[],
|
||||
deps_ready=True,
|
||||
sso_unverified=False,
|
||||
clean_teardown=True,
|
||||
no_secret_leak=True,
|
||||
finished_ts=1234.0,
|
||||
)
|
||||
# stateful, functional pass, no SSO surface, no repo-local → caps at L4
|
||||
# all four essential rungs pass → full climb to L4 (the top), no cap
|
||||
assert data["level"] == 4
|
||||
assert "L5" in data["level_cap_reason"]
|
||||
assert data["level_cap_reason"] == ""
|
||||
assert data["recipe"] == "hedgedoc"
|
||||
assert data["ref"] == "deadbeefcafe"
|
||||
assert data["flags"] == {"clean_teardown": True, "no_secret_leak": True}
|
||||
@ -246,9 +191,6 @@ def test_build_results_capped_at_L1_on_upgrade_fail(tmp_path):
|
||||
records=recs,
|
||||
results=_results(upgrade="fail"),
|
||||
backup_capable=True,
|
||||
declared=[],
|
||||
deps_ready=True,
|
||||
sso_unverified=False,
|
||||
clean_teardown=True,
|
||||
no_secret_leak=True,
|
||||
finished_ts=0.0,
|
||||
@ -266,8 +208,6 @@ def _rungs(**kw):
|
||||
"upgrade": "pass",
|
||||
"backup_restore": "pass",
|
||||
"functional": "pass",
|
||||
"integration": "na",
|
||||
"recipe_local": "na",
|
||||
}
|
||||
base.update(kw)
|
||||
return base
|
||||
@ -276,16 +216,16 @@ def _rungs(**kw):
|
||||
def test_skips_intentional_vs_unintentional():
|
||||
rungs = _rungs(backup_restore="na", functional="na")
|
||||
sk = R.skips(rungs, {"backup_restore": "stateless static server"})
|
||||
# backup_restore is declared (intentional, with reason); everything else skipped is unintentional.
|
||||
# backup_restore is declared (intentional, with reason); functional skipped but not declared.
|
||||
assert sk["intentional"] == {"backup_restore": "stateless static server"}
|
||||
assert sk["unintentional"] == ["functional", "integration", "recipe_local"]
|
||||
assert sk["unintentional"] == ["functional"]
|
||||
|
||||
|
||||
def test_skips_none_declared_all_unintentional():
|
||||
rungs = _rungs(backup_restore="na")
|
||||
sk = R.skips(rungs, None)
|
||||
assert sk["intentional"] == {}
|
||||
assert sk["unintentional"] == ["backup_restore", "integration", "recipe_local"]
|
||||
assert sk["unintentional"] == ["backup_restore"]
|
||||
|
||||
|
||||
def test_skips_declaration_only_counts_when_actually_skipped():
|
||||
@ -323,17 +263,10 @@ def test_build_results_threads_expected_na(tmp_path):
|
||||
records=recs,
|
||||
results=_results(backup="skip", restore="skip"), # custom=pass (default) → functional pass
|
||||
backup_capable=False, # no backupbot label → backup_restore skipped (N/A)
|
||||
declared=[],
|
||||
deps_ready=True,
|
||||
sso_unverified=False,
|
||||
clean_teardown=True,
|
||||
no_secret_leak=True,
|
||||
finished_ts=0.0,
|
||||
expected_na={
|
||||
"backup_restore": "stateless static file server",
|
||||
"integration": "no SSO surface",
|
||||
"recipe_local": "no upstream tests/",
|
||||
},
|
||||
expected_na={"backup_restore": "stateless static file server"},
|
||||
)
|
||||
# backup_restore skip still caps at L2 (never inflates) — even though functional passes above it,
|
||||
# the skip caps the climb — but it's the declared (intentional) rung that capped.
|
||||
@ -342,7 +275,7 @@ def test_build_results_threads_expected_na(tmp_path):
|
||||
assert data["level_cap_rung"] == "backup_restore"
|
||||
assert data["rungs"]["functional"] == "pass"
|
||||
assert data["skips"]["intentional"]["backup_restore"] == "stateless static file server"
|
||||
assert data["skips"]["unintentional"] == [] # every skip accounted for → fully clean
|
||||
assert data["skips"]["unintentional"] == [] # backup_restore declared; functional passed → clean
|
||||
|
||||
|
||||
def test_write_results_roundtrip(tmp_path):
|
||||
|
||||
Reference in New Issue
Block a user