feat(harness): declare intentional N/A tiers + custom-html-tiny functional test #6

Merged

autonomic-bot merged 7 commits from feat/expected-na-and-tiny-functional into main 2026-06-09 03:12:12 +00:00

Conversation 2 Commits 7 Files Changed 10 +392 -187

autonomic-bot commented 2026-06-09 02:12:42 +00:00

Owner

Copy Link

What

Two operator-requested changes, after noticing custom-html-tiny PR #6 had neither backup/restore nor functional coverage:

1. Declare intentional N/A tiers (vs. accidentally-missing coverage)

A recipe can now declare in recipe_meta.py:

EXPECTED_NA = {"backup_restore": "stateless static file server: ...why..."}

• N/A still caps the level — the harness never claims a rung it didn't verify (cardinal "never inflate" invariant, level.py untouched). This only explains the cap: results.json gains level_cap_intent ("intentional · <reason>") and an na block classifying every N/A rung as declared vs undeclared.
• An undeclared N/A on a gap-sensitive rung (backup_restore, functional) is surfaced as a possible coverage gap; a stale EXPECTED_NA (declared N/A but the tier actually ran) is surfaced too. Structurally-optional rungs (upgrade/integration/recipe_local) are recorded but never flagged.
• All non-blocking (R7): the summary card shows the clause; the CI log prints ⚠ coverage: / ⚠ stale EXPECTED_NA: lines.
• results.classify_na / cap_intent are pure + unit-tested.

custom-html-tiny declares backup_restore intentionally N/A (no backupbot.backup label — it serves an ephemeral, harness-seeded content volume with no persistent data).

2. custom-html-tiny functional test

Writes a random file into the served content volume (via the volume mountpoint, like install_steps.sh, since the static-web-server image is shell-less), asserts exact-byte round-trip + a real 404 on a missing path — proving it actually serves the volume, not a 200-everything fallback.

Verification

• cc-ci-run -m pytest tests/unit/test_results.py tests/unit/test_level.py tests/unit/test_card.py -q → 41 passed cold.
• Live harness run (install,custom) on custom-html-tiny: install + the new functional test PASS; results.json correctly classifies backup_restore as intent: declared, gaps: [], stale_declared: [].

NEVER merges — operator review. @notplants

🤖 Generated with Claude Code

## What Two operator-requested changes, after noticing `custom-html-tiny` PR #6 had neither backup/restore nor functional coverage: ### 1. Declare *intentional* N/A tiers (vs. accidentally-missing coverage) A recipe can now declare in `recipe_meta.py`: ```python EXPECTED_NA = {"backup_restore": "stateless static file server: ...why..."} ``` - **N/A still caps the level** — the harness never claims a rung it didn't verify (cardinal "never inflate" invariant, level.py untouched). This only **explains** the cap: `results.json` gains `level_cap_intent` (`"intentional · <reason>"`) and an `na` block classifying every N/A rung as `declared` vs `undeclared`. - An **undeclared** N/A on a *gap-sensitive* rung (`backup_restore`, `functional`) is surfaced as a **possible coverage gap**; a **stale** `EXPECTED_NA` (declared N/A but the tier actually ran) is surfaced too. Structurally-optional rungs (`upgrade`/`integration`/`recipe_local`) are recorded but never flagged. - All **non-blocking** (R7): the summary card shows the clause; the CI log prints `⚠ coverage:` / `⚠ stale EXPECTED_NA:` lines. - `results.classify_na` / `cap_intent` are pure + unit-tested. `custom-html-tiny` declares `backup_restore` intentionally N/A (no `backupbot.backup` label — it serves an ephemeral, harness-seeded content volume with no persistent data). ### 2. `custom-html-tiny` functional test Writes a random file into the served `content` volume (via the volume mountpoint, like `install_steps.sh`, since the static-web-server image is shell-less), asserts **exact-byte round-trip + a real 404** on a missing path — proving it actually serves the volume, not a 200-everything fallback. ## Verification - `cc-ci-run -m pytest tests/unit/test_results.py tests/unit/test_level.py tests/unit/test_card.py -q` → **41 passed** cold. - Live harness run (`install,custom`) on `custom-html-tiny`: install + the new functional test **PASS**; `results.json` correctly classifies `backup_restore` as `intent: declared`, `gaps: []`, `stale_declared: []`. NEVER merges — operator review. @notplants 🤖 Generated with [Claude Code](https://claude.com/claude-code)

autonomic-bot added 2 commits 2026-06-09 02:12:43 +00:00

feat(harness): declare intentional N/A tiers + custom-html-tiny functional test ... 3b0a3d14ea

Two changes the operator asked for after noticing custom-html-tiny PR #6 has no
backup/restore or functional coverage:

1) Intentional-vs-accidental N/A. A recipe can now declare
   recipe_meta.EXPECTED_NA = {rung: reason} to mark a tier as deliberately not
   applicable (e.g. a stateless static server has no backup surface). N/A still
   caps the level — the harness never claims a rung it did not verify — but the
   run is now annotated 'intentional · <reason>' instead of being indistinguishable
   from a forgotten test. An *undeclared* N/A on a gap-sensitive rung
   (backup_restore, functional) is surfaced as a 'possible coverage gap', and a
   stale EXPECTED_NA (declared N/A but actually exercised) is surfaced too. All
   non-blocking (R7): results.json gains level_cap_intent + an  block, the
   summary card shows the clause, and the CI log prints the gap/stale warnings.
   (results.classify_na/cap_intent are pure + unit-tested; level.py untouched.)

   custom-html-tiny declares backup_restore intentionally N/A.

2) custom-html-tiny functional test: writes a random file into the served content
   volume (via the volume mountpoint, like install_steps.sh, since the SWS image
   is shell-less), asserts exact-byte round-trip + a real 404 on a missing path —
   proving the static-web-server actually serves the volume, not a 200-everything
   fallback.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

test: representative expected_na scenario (functional covered, backup declared-N/A) ... f3a1ad5388

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

autonomic-bot added 1 commit 2026-06-09 02:26:46 +00:00

feat(card): badge differentiates expected vs unexpected skip ... d733e2c4ca

The level badge gains a third segment derived from level_cap_intent:
- amber 'gap?'   when the climb was capped by an UNDECLARED gap-sensitive N/A
                 (backup_restore / functional) — a likely-missing test (unexpected skip)
- muted 'expected' when capped by a DECLARED intentional N/A (reviewed, nothing to fix)
- nothing extra for a clean cap, a full climb, or a real failure.

Font-safe text labels (no emoji) so the SVG renders headless/anywhere. Badge
never inflates — it only annotates the cap the level already reflects.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

autonomic-bot added 1 commit 2026-06-09 02:36:54 +00:00

refactor: simplify to a list of intentionally-skipped rungs ... b3ab68a9dd

Per operator: drop the gap-sensitivity / cap-intent-clause / stale-detection
machinery. Model is now dead simple — recipe_meta.EXPECTED_NA = {rung: reason}
lists the rungs a recipe intentionally skips; ANY rung skipped (N/A) and not in
that list is unintentional.

results.json: replace the 'na' block + level_cap_intent with
  skips: { intentional: {rung: reason}, unintentional: [rung] }
plus level_cap_rung (which rung capped). Badge/card derive intentional-vs-
unintentional from whether the capping rung is in the intentional list. Skips
still cap the level (never inflate). custom-html-tiny lists all three rungs it
intentionally skips (backup_restore, integration, recipe_local).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

autonomic-bot commented 2026-06-09 02:39:21 +00:00

Author

Owner

Copy Link

New level badge — intentional vs unintentional skip

Live comparison (rendered from this branch's code): https://report.ci.commoninternet.net/badge-preview/index.html

Same recipe (custom-html-tiny), same level-2 cap on backup_restore — the only difference is whether the skip is declared in recipe_meta.EXPECTED_NA:

case	badge	card cap line
intentional (declared)	cc-ci │ level 2 │ expected (muted)	L3 backup/restore N/A · intentional: stateless static file server…
unintentional (not declared)	cc-ci │ level 2 │ gap? (amber)	L3 backup/restore N/A · unintentional skip (no EXPECTED_NA — add a test or declare it)

The level itself is unchanged in both cases — a skip always caps the climb (the harness never claims a rung it didn't verify). The third badge segment + card clause only label why the cap happened, derived from the simple rule: a skipped rung is intentional iff it's listed in EXPECTED_NA, otherwise it's an unintentional gap.

## New level badge — intentional vs unintentional skip Live comparison (rendered from this branch's code): **https://report.ci.commoninternet.net/badge-preview/index.html** Same recipe (`custom-html-tiny`), same level-2 cap on `backup_restore` — the only difference is whether the skip is declared in `recipe_meta.EXPECTED_NA`: | case | badge | card cap line | |---|---|---| | **intentional** (declared) | `cc-ci │ level 2 │ expected` (muted) | `L3 backup/restore N/A · intentional: stateless static file server…` | | **unintentional** (not declared) | `cc-ci │ level 2 │ gap?` (amber) | `L3 backup/restore N/A · unintentional skip (no EXPECTED_NA — add a test or declare it)` | The level itself is unchanged in both cases — a skip always caps the climb (the harness never claims a rung it didn't verify). The third badge segment + card clause only label *why* the cap happened, derived from the simple rule: a skipped rung is intentional iff it's listed in `EXPECTED_NA`, otherwise it's an unintentional gap.

autonomic-bot added 1 commit 2026-06-09 02:42:08 +00:00

feat(card): show skipped rungs as rows — INTENTIONAL SKIP (green) with reason below ... d20ad1e989

Per operator: intentional skips now render like a pass row but labelled
'INTENTIONAL SKIP' (muted green) with the declared reason on the line beneath;
unintentional skips render amber 'UNINTENTIONAL SKIP' with a prompt to add a test
or declare them. The cap line is back to just the level-cap reason (the per-rung
reason now lives in the rows). Labelled, so it never reads as a PASS.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

autonomic-bot added 1 commit 2026-06-09 02:42:59 +00:00

test(card): cover _skip_rows (intentional green / unintentional amber) ... 3980340727

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

autonomic-bot added 1 commit 2026-06-09 02:55:49 +00:00

refactor(level): four essential rungs only — integration & recipe-local are optional ... 46e2cdb93e

Per operator: the level ladder is now the FOUR essential rungs every recipe is
held to — install, upgrade (essential), backup/restore, functional (top = L4).
Integration (SSO/OIDC) and recipe-local are OPTIONAL capabilities: they no longer
appear as level rungs or skip rows and never cap the level. SSO is still enforced
for the run VERDICT (unchanged in run_recipe_ci.py); it just doesn't affect the
level. derive_rungs simplified accordingly (drops declared/deps/sso/repo-local
inputs). custom-html-tiny's EXPECTED_NA is back to just backup_restore.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

autonomic-bot commented 2026-06-09 02:57:52 +00:00

Author

Owner

Copy Link

Design as it now stands (supersedes the earlier comments)

After operator review the model was simplified considerably:

Level ladder = four essential rungs: install · upgrade · backup/restore · functional (top = L4). upgrade is essential. integration (SSO/OIDC) and recipe-local are OPTIONAL — they are no longer level rungs, never cap the level, and don't render as skip rows. (SSO is still enforced for the run verdict in run_recipe_ci.py; it just no longer affects the level.)

Intentional vs unintentional skips: a recipe lists the rungs it intentionally skips, with reasons, in recipe_meta.EXPECTED_NA = {rung: reason}. Any essential rung skipped (N/A) and not in that list is unintentional. A skip always caps the level (the harness never claims a rung it didn't verify) — this only labels why. results.json carries skips: {intentional:{rung:reason}, unintentional:[rung]} + level_cap_rung.

Display (live: https://report.ci.commoninternet.net/badge-preview/index.html):

• Card: skipped rungs render like pass rows but say INTENTIONAL SKIP (green ⊘, reason beneath) or UNINTENTIONAL SKIP (amber, prompt to declare/add).
• Badge: third segment expected (muted) for an intentional cap, gap? (amber) for an unintentional one.

custom-html-tiny: declares only backup_restore intentionally skipped (stateless static server); gets install + upgrade + the new functional serve test; caps at L2.

41 → 49 unit tests pass cold. NEVER merges by the harness — @notplants.

## Design as it now stands (supersedes the earlier comments) After operator review the model was simplified considerably: **Level ladder = four essential rungs:** `install · upgrade · backup/restore · functional` (top = **L4**). `upgrade` is essential. **integration (SSO/OIDC) and recipe-local are OPTIONAL** — they are no longer level rungs, never cap the level, and don't render as skip rows. (SSO is still enforced for the run *verdict* in `run_recipe_ci.py`; it just no longer affects the level.) **Intentional vs unintentional skips:** a recipe lists the rungs it intentionally skips, with reasons, in `recipe_meta.EXPECTED_NA = {rung: reason}`. Any essential rung skipped (N/A) and **not** in that list is *unintentional*. A skip always caps the level (the harness never claims a rung it didn't verify) — this only labels *why*. results.json carries `skips: {intentional:{rung:reason}, unintentional:[rung]}` + `level_cap_rung`. **Display** (live: https://report.ci.commoninternet.net/badge-preview/index.html): - Card: skipped rungs render like pass rows but say **INTENTIONAL SKIP** (green ⊘, reason beneath) or **UNINTENTIONAL SKIP** (amber, prompt to declare/add). - Badge: third segment `expected` (muted) for an intentional cap, `gap?` (amber) for an unintentional one. **custom-html-tiny:** declares only `backup_restore` intentionally skipped (stateless static server); gets install + upgrade + the new functional serve test; caps at L2. 41 → 49 unit tests pass cold. NEVER merges by the harness — @notplants.

autonomic-bot merged commit c51cd84159 into main 2026-06-09 03:12:12 +00:00

autonomic-bot referenced this issue from a commit 2026-06-09 03:12:12 +00:00

feat(harness): intentional skips + custom-html-tiny functional test; 4-rung ladder (#6)

Sign in to join this conversation.

Reviewers

No Reviewers

Labels

No items

No Label

Milestone

No items

No Milestone

Projects

Clear projects

No project

Assignees

Clear assignees

autonomic autonomic-bot notplants trav

No Assignees

1 Participants

Notifications

Subscribe

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: recipe-maintainers/cc-ci#6

No description provided.