plan(lvl5): N/A split — intentional skip climbs, unintentional (unverified) blocks
Operator refinement: only declared/structural skips (not backup-capable, no previous version) let the climb continue; a rung that should have run but didn't (infra error, abra missing, tier abort, timeout) blocks the level at the last verified rung. Every N/A source in derive_rungs gets an explicit classification (DECISIONS.md, adversary-reviewed); unclassifiable defaults to unverified. Unit tests + one synthesized tier-abort run prove the rule.
This commit is contained in:
@@ -416,3 +416,9 @@ session cc-ci-orchestrator-stale can be killed; recipe-mirrors org still private
|
|||||||
- Deliberate override of Phase-3 "N/A caps" stance — to be recorded in DECISIONS.md by
|
- Deliberate override of Phase-3 "N/A caps" stance — to be recorded in DECISIONS.md by
|
||||||
the loops. Before/after level table for all recipes required so the Adversary can
|
the loops. Before/after level table for all recipes required so the Adversary can
|
||||||
attribute every level shift to the rule change.
|
attribute every level shift to the rule change.
|
||||||
|
|
||||||
|
## 2026-06-11 ~02:00 — lvl5 refinement: intentional vs unintentional N/A
|
||||||
|
- Operator: an N/A rung only skips if it's an INTENTIONAL skip (declared/structural:
|
||||||
|
not backup-capable, no upgrade target). UNINTENTIONAL N/A (infra error, missing tool,
|
||||||
|
aborted tier = unverified) blocks — the level cannot be above an unverified rung.
|
||||||
|
Statuses now {pass, fail, skip, unver}; unclassifiable N/A defaults to unver.
|
||||||
|
|||||||
@@ -7,22 +7,36 @@
|
|||||||
upgrade, backup/restore, functional). The existing four rungs' meanings are UNCHANGED.
|
upgrade, backup/restore, functional). The existing four rungs' meanings are UNCHANGED.
|
||||||
|
|
||||||
2. **Remove the "capping" concept entirely.** The operator finds cap/cap_reason confusing.
|
2. **Remove the "capping" concept entirely.** The operator finds cap/cap_reason confusing.
|
||||||
New level semantics (operator-decided 2026-06-11, explicit Q&A):
|
New level semantics (operator-decided 2026-06-11, explicit Q&A + refinement):
|
||||||
**level = the highest rung that PASSED, where every rung below it is either "pass" or
|
**level = the highest rung that PASSED, where every rung below it is "pass" or an
|
||||||
"na" — N/A rungs are SKIPPED (they no longer stop the climb), a real FAIL still
|
INTENTIONAL skip.** A real FAIL blocks. A not-applicable rung is split in two:
|
||||||
blocks.** Formally: `level = max i such that rung_i == "pass" and all j < i have
|
- **Intentional skip (declared/structural):** the rung genuinely does not apply to
|
||||||
status in {"pass", "na"}`; 0 if no such i.
|
this recipe — e.g. not backup-capable (declared), only one published version exists
|
||||||
|
(no upgrade target). These are SKIPPED: they do not stop the climb.
|
||||||
|
- **Unintentional N/A (unverified):** the rung SHOULD have run but didn't — infra
|
||||||
|
error, missing tool (abra absent), harness exception, prior-stage abort, timeout.
|
||||||
|
**The level cannot rise above an unverified rung** — it blocks exactly like a fail
|
||||||
|
(the rungs below it still count).
|
||||||
|
Formally, with statuses {pass, fail, skip (intentional), unver (unintentional)}:
|
||||||
|
`level = max i such that rung_i == "pass" and all j < i have status in {"pass","skip"}`;
|
||||||
|
0 if no such i.
|
||||||
- install ✔, upgrade ✘, backup ✔, functional ✔, lint ✔ → **level 1** (fail blocks)
|
- install ✔, upgrade ✘, backup ✔, functional ✔, lint ✔ → **level 1** (fail blocks)
|
||||||
- install ✔, upgrade ✔, backup N/A, functional ✔, lint ✔ → **level 5** (N/A skipped —
|
- install ✔, upgrade ✔, backup skip(not capable), functional ✔, lint ✔ → **level 5**
|
||||||
previously this capped at 2; that was the confusing part)
|
(intentional skip — previously this capped at 2; that was the confusing part)
|
||||||
- all four ✔, lint N/A (e.g. abra missing) → **level 4** (an N/A rung is never EARNED,
|
- install ✔, upgrade ✔, backup UNVER (harness error), functional ✔, lint ✔ →
|
||||||
it just doesn't block the ones above)
|
**level 2** (unverified blocks — we cannot claim what we didn't check)
|
||||||
|
- all four ✔, lint unver (abra missing) → **level 4** (an unverified top rung is
|
||||||
|
simply not earned)
|
||||||
|
**Every N/A source in derive_rungs must be explicitly classified** intentional vs
|
||||||
|
unintentional, from declared recipe meta / structural facts vs runtime failure —
|
||||||
|
the classification table goes in DECISIONS.md and is Adversary-reviewed. Default for
|
||||||
|
an unclassifiable N/A is UNINTENTIONAL (conservative).
|
||||||
The words "cap", "capped", "cap_reason" disappear from code, schema, card, dashboard,
|
The words "cap", "capped", "cap_reason" disappear from code, schema, card, dashboard,
|
||||||
and docs. The per-rung ✔/✘/— table remains everywhere it exists today, so nothing is
|
and docs. The per-rung table remains everywhere it exists today (now distinguishing
|
||||||
hidden — the table is now the SOLE carrier of "why isn't this level higher".
|
✔ / ✘ / intentional-skip / unverified), so nothing is hidden — the table is the SOLE
|
||||||
NOTE this is a deliberate operator override of the old "N/A caps so the level never
|
carrier of "why isn't this level higher". Record the whole semantics change in
|
||||||
overstates" stance from Phase 3: the number now reads "how far did it get", and the
|
DECISIONS.md (it deliberately replaces the Phase-3 "N/A caps" stance; the
|
||||||
rung table + run verdict carry the completeness story. Record this in DECISIONS.md.
|
unverified-blocks rule preserves its honest core: never claim what wasn't checked).
|
||||||
|
|
||||||
State files (phase-namespaced): `STATUS-lvl5.md`, `BACKLOG-lvl5.md`, `REVIEW-lvl5.md`,
|
State files (phase-namespaced): `STATUS-lvl5.md`, `BACKLOG-lvl5.md`, `REVIEW-lvl5.md`,
|
||||||
`JOURNAL-lvl5.md`. DECISIONS.md shared (append).
|
`JOURNAL-lvl5.md`. DECISIONS.md shared (append).
|
||||||
@@ -69,9 +83,9 @@ State files (phase-namespaced): `STATUS-lvl5.md`, `BACKLOG-lvl5.md`, `REVIEW-lvl
|
|||||||
verdict, and must be time-bounded + best-effort in execution (a wedged lint can never
|
verdict, and must be time-bounded + best-effort in execution (a wedged lint can never
|
||||||
hang a run — hard timeout, ~60s class).
|
hang a run — hard timeout, ~60s class).
|
||||||
5. **No N/A escape hatch by default:** every recipe can be linted, so the rung is
|
5. **No N/A escape hatch by default:** every recipe can be linted, so the rung is
|
||||||
pass/fail in practice (keep "na" handling for totality, e.g. abra binary missing →
|
pass/fail in practice. When lint can't run (abra binary missing, timeout) the status
|
||||||
"na" + loud log — never silently "pass"; per the new semantics an N/A lint rung is
|
is **unver** + loud log — never silently "pass", never an intentional skip; the rung
|
||||||
simply not earned, so the level stays 4).
|
is not earned and the level stays ≤ 4.
|
||||||
6. **De-cap implementation (mission item 2):** `compute_level()` reimplemented to the
|
6. **De-cap implementation (mission item 2):** `compute_level()` reimplemented to the
|
||||||
new formal rule; `cap_reason`/`capped` deleted from level.py, derive_rungs/results.json
|
new formal rule; `cap_reason`/`capped` deleted from level.py, derive_rungs/results.json
|
||||||
schema, card (the "capped: …"/"full clean climb — top level (4)" line at card.py:~246
|
schema, card (the "capped: …"/"full clean climb — top level (4)" line at card.py:~246
|
||||||
@@ -97,9 +111,12 @@ State files (phase-namespaced): `STATUS-lvl5.md`, `BACKLOG-lvl5.md`, `REVIEW-lvl
|
|||||||
**P1 — Ladder + plumbing.** level.py: new rung + the de-capped `compute_level()` per
|
**P1 — Ladder + plumbing.** level.py: new rung + the de-capped `compute_level()` per
|
||||||
§2 item 2 (delete cap_reason/capped); lint executor (new `harness/lint.py` or a clean
|
§2 item 2 (delete cap_reason/capped); lint executor (new `harness/lint.py` or a clean
|
||||||
home in abra.py) with timeout, artifact capture, mirror-context handling per §2.3;
|
home in abra.py) with timeout, artifact capture, mirror-context handling per §2.3;
|
||||||
derive_rungs wiring; results schema. Unit tests for: full climb = 5; fail-blocks
|
derive_rungs wiring (incl. the intentional-vs-unintentional classification of every N/A
|
||||||
(upgrade ✘ → L1 even with higher passes); N/A-skip (backup na + functional ✔ → L4+);
|
source, §2 item 2); results schema. Unit tests for: full climb = 5; fail-blocks
|
||||||
lint na → stays 4; old-artifact rendering; mirror-filter decision (if any).
|
(upgrade ✘ → L1 even with higher passes); intentional-skip climbs (backup skip +
|
||||||
|
functional ✔ → L4+); unverified-blocks (backup unver + functional ✔ → L2); lint unver →
|
||||||
|
stays 4; unclassifiable N/A defaults to unver; old-artifact rendering; mirror-filter
|
||||||
|
decision (if any).
|
||||||
|
|
||||||
**P2 — Presentation.** Card, dashboard, badge, docs — a level-5 color/legend that reads
|
**P2 — Presentation.** Card, dashboard, badge, docs — a level-5 color/legend that reads
|
||||||
as "above functional". Regenerate anything generated.
|
as "above functional". Regenerate anything generated.
|
||||||
@@ -119,8 +136,10 @@ violation is fine), ≥1 recipe demonstrating the N/A-skip (formerly capped by a
|
|||||||
now climbing past it), ≥2 runs via the drone `!testme` path showing the rung on a real
|
now climbing past it), ≥2 runs via the drone `!testme` path showing the rung on a real
|
||||||
PR, plus the canary suite green — the bad canaries must still land at their designed
|
PR, plus the canary suite green — the bad canaries must still land at their designed
|
||||||
levels under the NEW formula (re-derive what those designed levels are; backup/restore
|
levels under the NEW formula (re-derive what those designed levels are; backup/restore
|
||||||
fail still blocks). Card + dashboard visually verified (Read the PNGs/SVG output), and
|
fail still blocks). The unverified-blocks rule is proven by unit test + one synthesized
|
||||||
the §2 item 9 before/after level table completed for all enrolled recipes.
|
run (induce a tier abort and check the level stays below the unverified rung). Card +
|
||||||
|
dashboard visually verified (Read the PNGs/SVG output), and the §2 item 9 before/after
|
||||||
|
level table completed for all enrolled recipes.
|
||||||
|
|
||||||
## 4. Gates
|
## 4. Gates
|
||||||
|
|
||||||
@@ -159,8 +178,9 @@ Builder writes `## DONE` to STATUS-lvl5.md.
|
|||||||
|
|
||||||
The new level system live on main and visible end-to-end (results.json → card →
|
The new level system live on main and visible end-to-end (results.json → card →
|
||||||
dashboard → badge): L5 = abra recipe lint on the tested ref, capping concept fully
|
dashboard → badge): L5 = abra recipe lint on the tested ref, capping concept fully
|
||||||
removed (level = highest passed rung with all lower rungs pass-or-N/A; N/A skips, fail
|
removed (level = highest passed rung with all lower rungs pass-or-intentional-skip;
|
||||||
blocks; no cap/cap_reason anywhere), all enrolled recipes linted and dispositioned with
|
declared skips climb, fails AND unverified rungs block; no cap/cap_reason anywhere),
|
||||||
|
all enrolled recipes linted and dispositioned with
|
||||||
the before/after level table adversary-checked, ≥1 real L5 + ≥1 lint-blocked L4 + ≥1
|
the before/after level table adversary-checked, ≥1 real L5 + ≥1 lint-blocked L4 + ≥1
|
||||||
N/A-skip climb demonstrated through real CI including the drone path, old artifacts
|
N/A-skip climb demonstrated through real CI including the drone path, old artifacts
|
||||||
unharmed, M1+M2 fresh Adversary PASSes, no verdict or duration regressions.
|
unharmed, M1+M2 fresh Adversary PASSes, no verdict or duration regressions.
|
||||||
|
|||||||
Reference in New Issue
Block a user