feat(harness): declare intentional N/A tiers + custom-html-tiny functional test
Some checks failed
continuous-integration/drone/push Build is failing
Some checks failed
continuous-integration/drone/push Build is failing
Two changes the operator asked for after noticing custom-html-tiny PR #6 has no backup/restore or functional coverage: 1) Intentional-vs-accidental N/A. A recipe can now declare recipe_meta.EXPECTED_NA = {rung: reason} to mark a tier as deliberately not applicable (e.g. a stateless static server has no backup surface). N/A still caps the level — the harness never claims a rung it did not verify — but the run is now annotated 'intentional · <reason>' instead of being indistinguishable from a forgotten test. An *undeclared* N/A on a gap-sensitive rung (backup_restore, functional) is surfaced as a 'possible coverage gap', and a stale EXPECTED_NA (declared N/A but actually exercised) is surfaced too. All non-blocking (R7): results.json gains level_cap_intent + an block, the summary card shows the clause, and the CI log prints the gap/stale warnings. (results.classify_na/cap_intent are pure + unit-tested; level.py untouched.) custom-html-tiny declares backup_restore intentionally N/A. 2) custom-html-tiny functional test: writes a random file into the served content volume (via the volume mountpoint, like install_steps.sh, since the SWS image is shell-less), asserts exact-byte round-trip + a real 404 on a missing path — proving the static-web-server actually serves the volume, not a 200-everything fallback. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
87
tests/custom-html-tiny/functional/test_serves_content.py
Normal file
87
tests/custom-html-tiny/functional/test_serves_content.py
Normal file
@ -0,0 +1,87 @@
|
||||
"""custom-html-tiny — recipe-specific functional test (static-web-server).
|
||||
|
||||
Proves the deployed static-web-server is *actually serving files from its `content` volume* with real
|
||||
file-server semantics, not merely returning 200 from a Traefik fallback or a generic stub:
|
||||
|
||||
1. exact-byte round-trip — write a uniquely-named file with random content into the served volume,
|
||||
fetch it over HTTPS, and assert the bytes come back verbatim. Non-vacuous: the content is random
|
||||
per run, so only a server that reads this file off the volume can pass.
|
||||
2. real 404 — a random non-existent path returns 404, proving directory/file semantics (a
|
||||
200-everything stub or mis-routed host would not 404).
|
||||
|
||||
The recipe's image (joseluisq/static-web-server) is shell-less (scratch-based) and its content volume
|
||||
is seeded via the install_steps.sh host-mountpoint mechanism — so this test writes its probe file the
|
||||
same way (resolve the swarm volume's mountpoint with `docker volume inspect`, write directly) rather
|
||||
than `docker exec`-ing in a container that has no shell.
|
||||
|
||||
Runs in the custom tier against the shared post-install deployment (the `live_app` fixture is its
|
||||
per-run domain). Mirrors install_steps.sh: the app's content volume is named `<stack>_content`, where
|
||||
`stack` is the domain with dots replaced by underscores; HTTP_SUBDIR is empty, so the volume root is
|
||||
served at `/`.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import contextlib
|
||||
import os
|
||||
import ssl
|
||||
import subprocess
|
||||
import urllib.error
|
||||
import urllib.request
|
||||
import uuid
|
||||
|
||||
|
||||
def _served_dir(domain: str) -> str:
|
||||
"""Host mountpoint of the app's served `content` volume (same naming as install_steps.sh)."""
|
||||
vol = f"{domain.replace('.', '_')}_content"
|
||||
out = subprocess.run(
|
||||
["docker", "volume", "inspect", vol, "--format", "{{.Mountpoint}}"],
|
||||
capture_output=True,
|
||||
text=True,
|
||||
check=True,
|
||||
)
|
||||
mountpoint = out.stdout.strip()
|
||||
assert mountpoint, f"could not resolve mountpoint for volume {vol!r}"
|
||||
return mountpoint
|
||||
|
||||
|
||||
def _get(url: str) -> tuple[int, bytes]:
|
||||
"""GET the URL; return (status, body). A 4xx/5xx is returned, not raised (we assert on the code).
|
||||
TLS verification is relaxed: the served wildcard cert is validated separately by the infra check;
|
||||
here we care only about the app's response."""
|
||||
ctx = ssl.create_default_context()
|
||||
ctx.check_hostname = False
|
||||
ctx.verify_mode = ssl.CERT_NONE
|
||||
try:
|
||||
with urllib.request.urlopen(url, timeout=20, context=ctx) as resp:
|
||||
return resp.status, resp.read()
|
||||
except urllib.error.HTTPError as e:
|
||||
return e.code, e.read()
|
||||
|
||||
|
||||
def test_static_file_roundtrip_and_404(live_app):
|
||||
"""Write a random file into the served volume → fetch it → bytes match; and a missing path 404s."""
|
||||
served = _served_dir(live_app)
|
||||
token = uuid.uuid4().hex
|
||||
name = f"ccci-probe-{token}.txt"
|
||||
body = f"cc-ci-functional-{token}\n".encode()
|
||||
path = os.path.join(served, name)
|
||||
with open(path, "wb") as fh:
|
||||
fh.write(body)
|
||||
try:
|
||||
status, got = _get(f"https://{live_app}/{name}")
|
||||
assert status == 200, f"served probe file returned {status} (expected 200)"
|
||||
assert got == body, (
|
||||
f"content round-trip mismatch: served {got!r}, wrote {body!r} "
|
||||
"(static-web-server not serving the content volume?)"
|
||||
)
|
||||
|
||||
# A random non-existent path must 404 — proves real static-file semantics, distinguishing a
|
||||
# working server from a 200-everything stub or a mis-routed Traefik fallback.
|
||||
miss_status, _ = _get(f"https://{live_app}/ccci-missing-{uuid.uuid4().hex}.txt")
|
||||
assert miss_status == 404, (
|
||||
f"missing path returned {miss_status} (expected 404 — generic 200-returner / mis-route?)"
|
||||
)
|
||||
finally:
|
||||
with contextlib.suppress(OSError):
|
||||
os.remove(path)
|
||||
Reference in New Issue
Block a user