feat(harness): declare intentional N/A tiers + custom-html-tiny functional test
Some checks failed
continuous-integration/drone/push Build is failing

Two changes the operator asked for after noticing custom-html-tiny PR #6 has no
backup/restore or functional coverage:

1) Intentional-vs-accidental N/A. A recipe can now declare
   recipe_meta.EXPECTED_NA = {rung: reason} to mark a tier as deliberately not
   applicable (e.g. a stateless static server has no backup surface). N/A still
   caps the level — the harness never claims a rung it did not verify — but the
   run is now annotated 'intentional · <reason>' instead of being indistinguishable
   from a forgotten test. An *undeclared* N/A on a gap-sensitive rung
   (backup_restore, functional) is surfaced as a 'possible coverage gap', and a
   stale EXPECTED_NA (declared N/A but actually exercised) is surfaced too. All
   non-blocking (R7): results.json gains level_cap_intent + an  block, the
   summary card shows the clause, and the CI log prints the gap/stale warnings.
   (results.classify_na/cap_intent are pure + unit-tested; level.py untouched.)

   custom-html-tiny declares backup_restore intentionally N/A.

2) custom-html-tiny functional test: writes a random file into the served content
   volume (via the volume mountpoint, like install_steps.sh, since the SWS image
   is shell-less), asserts exact-byte round-trip + a real 404 on a missing path —
   proving the static-web-server actually serves the volume, not a 200-everything
   fallback.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
autonomic-bot
2026-06-09 01:59:28 +00:00
parent f5a6f7196f
commit 3b0a3d14ea
6 changed files with 279 additions and 3 deletions

View File

@ -200,6 +200,7 @@ def _load_meta(recipe: str) -> dict:
for k in list(meta) + [
"BACKUP_CAPABLE",
"SKIP_GENERIC",
"EXPECTED_NA",
"OIDC_AT_INSTALL",
"READY_PROBE",
"UPGRADE_BASE_VERSION",
@ -1241,6 +1242,7 @@ def main() -> int:
no_secret_leak=True, # narrowed below by an actual scan of the serialised artifact
screenshot=screenshot_rel, # Phase 3 U1 (R4): relative PNG name iff capture succeeded
finished_ts=time.time(),
expected_na=meta.get("EXPECTED_NA"), # declared intentional-N/A map (recipe_meta)
)
# Real (if narrow) leak check: no known infra-secret value may appear in the artifact (R7).
blob = json.dumps(data)
@ -1252,11 +1254,29 @@ def main() -> int:
file=sys.stderr,
)
path = results_mod.write_results(data)
intent = data.get("level_cap_intent") or ""
print(
f"results.json written: {path} (level={data['level']}"
f"{'' + data['level_cap_reason'] if data['level_cap_reason'] else ''})",
f"{'' + data['level_cap_reason'] if data['level_cap_reason'] else ''}"
f"{' [' + intent + ']' if intent else ''})",
flush=True,
)
# Surface the intentional-vs-accidental N/A signal in the CI log (non-blocking, R7): a
# gap-sensitive rung that is N/A but undeclared is a possible coverage hole; a stale
# EXPECTED_NA declares a tier N/A that actually ran.
na = data.get("na", {})
for rung in na.get("gaps", []):
print(
f"⚠ coverage: rung '{rung}' is N/A but not declared intentional — add a test or "
f"declare it in tests/{recipe}/recipe_meta.py EXPECTED_NA = {{'{rung}': '<why>'}}.",
flush=True,
)
for rung in na.get("stale_declared", []):
print(
f"⚠ stale EXPECTED_NA: rung '{rung}' is declared N/A but was actually exercised "
f"(status={data['rungs'].get(rung)}) — remove it from recipe_meta.EXPECTED_NA.",
flush=True,
)
except Exception as e: # noqa: BLE001 — results assembly is cosmetic; never fail a run on it (R7)
print(
f"!! results.json assembly failed (non-fatal, verdict unaffected): {_scrub(str(e))}",