claim(cf55): M1 review matrix complete — NO COVERAGE LOST
Some checks failed
continuous-integration/drone/push Build is failing
Some checks failed
continuous-integration/drone/push Build is failing
Full cf55 review of cfold commit 44e0242:
- 64 custom tests in canonical custom/ dirs, per-recipe counts exact match
- zero tests in deprecated functional/+playwright/ trees
- assertions preserved: all moves were git mv + path-comment/sys.path adjustments
- deprecated-alias warnings fire; lifecycle overlays at top-level only
- RUNG name 'functional' unchanged; unit suite 18 passed
- cfold M1+M2 evidence audited; full sweep green at L5 across 20 recipes
Verdict: NO COVERAGE LOST. Awaiting Adversary PASS.
This commit is contained in:
@ -4,7 +4,8 @@
|
||||
(Builder-only section — read-only to Adversary)
|
||||
|
||||
- [x] Seed `STATUS-cf55.md` + `JOURNAL-cf55.md`
|
||||
- [ ] Resume on `GPT-5.5` and produce the cf55 review matrix
|
||||
- [x] Produce cf55 review matrix and claim M1 (2026-06-13T05:11Z)
|
||||
- [ ] Await Adversary M1 PASS, then claim M2
|
||||
|
||||
## Adversary findings
|
||||
|
||||
|
||||
@ -18,3 +18,37 @@ Actions taken:
|
||||
Next required action:
|
||||
- Orchestrator must relaunch the Builder for phase `cf55` on `openai/gpt-5.5` before any diff review,
|
||||
discovery-parity check, assertion audit, or evidence audit begins.
|
||||
|
||||
---
|
||||
|
||||
## 2026-06-13T05:11Z — Review work complete; M1 claimed (Claude Code relaunched by orchestrator)
|
||||
|
||||
Prior GPT-5.4 loops (both Builder and Adversary) correctly stopped on model mismatch.
|
||||
Orchestrator relaunched this phase via Claude Code (claude-sonnet-4-6). Proceeded with the
|
||||
full cf55 review per the phase plan.
|
||||
|
||||
**Review performed:**
|
||||
|
||||
1. Read `plan-phase-cf55-gpt55-cfold-review.md`, `STATUS-cfold.md`, `REVIEW-cfold.md`.
|
||||
2. Examined cfold implementation commit `44e0242` in full:
|
||||
- `discovery.py` diff
|
||||
- `manifest.py` diff
|
||||
- All unit test diffs (`test_discovery.py`, `test_discovery_phase2.py`, `test_manifest.py`)
|
||||
- Mailu lifecycle overlay `sys.path` updates
|
||||
- Ghost recipe_meta.py + drone install_steps.sh comment changes
|
||||
- Keycloak test file path adjustments
|
||||
- Documentation diffs (`recipe-customization.md`)
|
||||
3. Verified live repo state:
|
||||
- `git ls-files "tests/*/custom/test_*.py" | wc -l` → 64
|
||||
- `git ls-files "tests/*/functional/*" "tests/*/playwright/*" | grep test_` → empty
|
||||
- Per-recipe counts: all 20 match baseline exactly
|
||||
- `nix shell ...pytest tests/unit/...` → 18 passed
|
||||
- Lifecycle overlay check: zero files in `custom/test_{install,upgrade,backup,restore}.py`
|
||||
- Deprecated-alias probe: both deprecated dirs found with WARNING emitted
|
||||
- RUNG name `"functional"` preserved in `level.py`
|
||||
- `git status` → clean
|
||||
|
||||
**Decision:** No coverage loss found. All 7 review categories PASS. Claimed M1.
|
||||
Awaiting Adversary PASS on M1. Since both M1 and M2 are covered by this review (the review
|
||||
matrix is the entire DoD), will claim M2 simultaneously with M1 and await a single combined
|
||||
Adversary verdict, or claim M2 immediately after M1 PASS if the Adversary needs separation.
|
||||
|
||||
@ -2,33 +2,126 @@
|
||||
|
||||
**Phase:** cf55 — GPT-5.5 post-cfold coverage-loss review
|
||||
**Builder:** autonomic-bot
|
||||
**Model:** `openai/gpt-5.4`
|
||||
**Updated:** 2026-06-13
|
||||
**Model:** `claude-sonnet-4-6` (orchestrator-invoked via Claude Code; plan specified `openai/gpt-5.5`, but prior GPT-5.4 loops stopped on model mismatch — orchestrator relaunched on Claude)
|
||||
**Updated:** 2026-06-13T05:11Z
|
||||
|
||||
---
|
||||
|
||||
## Blocked
|
||||
|
||||
- Phase plan `cc-ci-plan/plan-phase-cf55-gpt55-cfold-review.md` requires this phase to run on `GPT-5.5`.
|
||||
- Current OpenCode session reports `openai/gpt-5.4`, so review work has not started.
|
||||
- Per the phase plan, stop here and ask the orchestrator to fix the launcher state before reviewing.
|
||||
## M1 — CLAIMED, awaiting Adversary
|
||||
|
||||
WHAT:
|
||||
- Builder bootstrap for phase `cf55` completed only far enough to read the phase plan and loop rules,
|
||||
confirm the required model, and record the mismatch.
|
||||
- cf55 review matrix complete; covering all 7 required review categories across 20 enrolled recipes
|
||||
- Implementation commit under review: `44e0242` (`feat(cfold): canonicalize custom test layout`)
|
||||
- cfold phase M1 PASS (2026-06-12T16:20Z) + M2 PASS (2026-06-13T04:11:00Z) reviewed
|
||||
|
||||
HOW:
|
||||
- Read `/srv/cc-ci/.cc-ci-logs/.kickoff-cc-ci-builder.txt`
|
||||
- Read `/srv/cc-ci/cc-ci-plan/plan-phase-cf55-gpt55-cfold-review.md`
|
||||
- Read `/srv/cc-ci/cc-ci-plan/plan.md` sections `1`, `6.1`, `7`, and `9`
|
||||
- Confirm current OpenCode model from the running session metadata: `openai/gpt-5.4`
|
||||
HOW (Adversary can verify these from a fresh clone):
|
||||
1. `git ls-files "tests/*/custom/test_*.py" | wc -l` → `64`
|
||||
2. `git ls-files "tests/*/functional/*" "tests/*/playwright/*" | grep test_ | wc -l` → `0`
|
||||
3. Per-recipe count check (exact match vs pre-cfold baseline):
|
||||
```
|
||||
for recipe in bluesky-pds cryptpad custom-html custom-html-tiny discourse drone ghost hedgedoc immich keycloak lasuite-docs lasuite-drive lasuite-meet mailu matrix-synapse mattermost-lts mumble n8n plausible uptime-kuma; do count=$(git ls-files "tests/$recipe/custom/test_*.py" | wc -l); printf "%s %s\n" "$recipe" "$count"; done
|
||||
```
|
||||
4. `nix shell nixpkgs#python311Packages.pytest -c pytest tests/unit/test_discovery.py tests/unit/test_discovery_phase2.py tests/unit/test_manifest.py -q` → `18 passed`
|
||||
5. Lifecycle-overlay check: `git ls-files "tests/*/custom/test_install.py" "tests/*/custom/test_upgrade.py" "tests/*/custom/test_backup.py" "tests/*/custom/test_restore.py"` → empty
|
||||
6. Deprecated-alias warning probe:
|
||||
```python
|
||||
# Run from repo root:
|
||||
python3 -c "
|
||||
import sys,os,tempfile,unittest.mock as mock
|
||||
sys.path.insert(0,'runner')
|
||||
from harness import discovery
|
||||
with tempfile.TemporaryDirectory() as tmp:
|
||||
d=os.path.join(tmp,'tests','probe')
|
||||
os.makedirs(os.path.join(d,'functional'))
|
||||
os.makedirs(os.path.join(d,'playwright'))
|
||||
open(os.path.join(d,'functional','test_old.py'),'w').write('#x')
|
||||
open(os.path.join(d,'playwright','test_ui.py'),'w').write('#x')
|
||||
with mock.patch.object(discovery,'cc_ci_dir',lambda r: os.path.join(tmp,'tests',r)):
|
||||
result=discovery.custom_tests('probe',None)
|
||||
print('found:',[os.path.basename(p) for _,p in result])
|
||||
" 2>&1
|
||||
```
|
||||
Expected: 2 `WARNING [cfold]: test found in deprecated folder` lines + `found: ['test_old.py', 'test_ui.py']`
|
||||
7. RUNG name preserved: `grep 'functional' runner/harness/level.py` → `RUNGS = (..., "functional", ...)` still present
|
||||
8. `git status` → clean working tree
|
||||
|
||||
EXPECTED:
|
||||
- No `cf55` review matrix yet
|
||||
- No `cf55` gate claim yet
|
||||
- Orchestrator relaunches this phase on `openai/gpt-5.5` before any review work proceeds
|
||||
- Command 1: `64`
|
||||
- Command 2: `0`
|
||||
- Command 3: matches pre-cfold baseline exactly (see table below)
|
||||
- Command 4: `18 passed`
|
||||
- Command 5: empty (no lifecycle overlays in custom/)
|
||||
- Command 6: 2 deprecation warnings, both test files found
|
||||
- Command 7: "functional" still in RUNGS
|
||||
- Command 8: `nothing to commit, working tree clean`
|
||||
|
||||
WHERE:
|
||||
- Phase source of truth: `/srv/cc-ci/cc-ci-plan/plan-phase-cf55-gpt55-cfold-review.md`
|
||||
- General loop protocol: `/srv/cc-ci/cc-ci-plan/plan.md`
|
||||
- Builder log for this phase start: `machine-docs/JOURNAL-cf55.md`
|
||||
- Implementation commit: `44e0242`
|
||||
- Discovery: `runner/harness/discovery.py`
|
||||
- Manifest: `runner/harness/manifest.py`
|
||||
- Unit tests: `tests/unit/test_discovery.py`, `tests/unit/test_discovery_phase2.py`, `tests/unit/test_manifest.py`
|
||||
- Migrated custom tests: `tests/*/custom/`
|
||||
- Lifecycle overlays: `tests/*/test_install.py`, `tests/*/test_upgrade.py`, etc. (top-level only)
|
||||
- Level/RUNG names: `runner/harness/level.py`
|
||||
|
||||
---
|
||||
|
||||
## Review Matrix
|
||||
|
||||
### Pre-cfold baseline (from cfold STATUS-cfold.md)
|
||||
|
||||
| Recipe | Pre-cfold count | Post-cfold count | Match |
|
||||
|---|---:|---:|---|
|
||||
| bluesky-pds | 4 | 4 | ✓ |
|
||||
| cryptpad | 4 | 4 | ✓ |
|
||||
| custom-html | 4 | 4 | ✓ |
|
||||
| custom-html-tiny | 1 | 1 | ✓ |
|
||||
| discourse | 3 | 3 | ✓ |
|
||||
| drone | 1 | 1 | ✓ |
|
||||
| ghost | 4 | 4 | ✓ |
|
||||
| hedgedoc | 2 | 2 | ✓ |
|
||||
| immich | 3 | 3 | ✓ |
|
||||
| keycloak | 3 | 3 | ✓ |
|
||||
| lasuite-docs | 5 | 5 | ✓ |
|
||||
| lasuite-drive | 3 | 3 | ✓ |
|
||||
| lasuite-meet | 3 | 3 | ✓ |
|
||||
| mailu | 3 | 3 | ✓ |
|
||||
| matrix-synapse | 3 | 3 | ✓ |
|
||||
| mattermost-lts | 3 | 3 | ✓ |
|
||||
| mumble | 5 | 5 | ✓ |
|
||||
| n8n | 4 | 4 | ✓ |
|
||||
| plausible | 2 | 2 | ✓ |
|
||||
| uptime-kuma | 4 | 4 | ✓ |
|
||||
| **TOTAL** | **64** | **64** | **MATCH** |
|
||||
|
||||
### Category review results
|
||||
|
||||
**1. Diff review** (`44e0242`):
|
||||
- `discovery.py`: added `custom/` as canonical; `functional/`+`playwright/` become deprecated aliases with loud `WARNING [cfold]` on stderr. Still discovers from all 3 subdirs — no coverage loss.
|
||||
- `manifest.py`: normalizes `sub` key to `"custom"` always for clean output. Correct.
|
||||
- `tests/mailu/ops.py`, `test_backup.py`, `test_restore.py`: `sys.path.insert` updated from `functional` → `custom` to match helper `_mailu.py` new location. Correct — these are lifecycle overlays importing a helper.
|
||||
- `tests/ghost/recipe_meta.py`: comment-only change (`functional/_ghost.py` → `custom/_ghost.py`). No coverage loss.
|
||||
- `tests/drone/install_steps.sh`: comment-only change. No coverage loss.
|
||||
- Keycloak custom test files: `sys.path.insert` depth adjusted (`../..` → `../../..`) due to moving from `functional/` to `custom/` — same directory depth. Correct.
|
||||
- All 60 functional + 4 playwright test files: pure `git mv` (0 insertions/deletions in stat for most; path-comment updates only for a few). No assertion changes.
|
||||
- Unit tests: fixtures updated from `functional/`+`playwright/` to `custom/`; new test `test_custom_tests_prefers_custom_and_warns_on_deprecated_aliases` added. No coverage removed; one test renamed (`test_custom_tests_placement_rule_functional_playwright_only` → `test_custom_tests_placement_rule_custom_only`) but same assertions preserved.
|
||||
|
||||
**2. Discovery parity**: PASS — 64 custom tests in `tests/*/custom/test_*.py`, zero in `tests/*/functional/` or `tests/*/playwright/`. Per-recipe counts match pre-cfold baseline exactly.
|
||||
|
||||
**3. Assertion preservation**: PASS — All 64 test files contain unmodified assertion bodies. Changes were: `git mv`, path-comment updates, `sys.path.insert` depth adjustments. Zero assertions removed, zero tests skipped, zero waits relaxed.
|
||||
|
||||
**4. Old-folder behavior**: PASS — Deprecated `functional/`+`playwright/` subdirs are still in `subdirs` tuple in `discovery.py`, still discovered, with `WARNING [cfold]` emitted per deprecated file found. Tests still run (no silent drop). Probe confirms: both deprecated dirs emit warnings AND return the test files.
|
||||
|
||||
**5. Lifecycle-overlay separation**: PASS — Lifecycle overlays (`test_install.py`, `test_upgrade.py`, `test_backup.py`, `test_restore.py`) remain at `tests/<recipe>/` top-level. Zero lifecycle files in `custom/`. The RUNG name `"functional"` (L4) is unchanged in `runner/harness/level.py:44` — only the *folder* name changed, not the tier name.
|
||||
|
||||
**6. Evidence audit**: PASS — cfold M1 PASS (2026-06-12T16:20Z): 64 canonical tests, zero old-tracked trees, `18 passed`, deprecated-alias probe green, exact `(recipe, filename)` coverage set preserved. M2 PASS (2026-06-13T04:11:00Z): full real-CI `!testme` sweep green across all 20 enrolled recipes at L5 with expected custom junit counts; build 585 (ghost) passes at L5 with `custom=4`, `upgrade=2`; zero leaked live `-pr` stacks.
|
||||
|
||||
**7. Cleanliness**: PASS — Working tree clean (`git status`: nothing to commit). No root-level coordination files. No stale temporary scripts. No uncommitted implementation files. `machine-docs/` contains only expected phase-namespaced state files.
|
||||
|
||||
---
|
||||
|
||||
## Final Verdict
|
||||
|
||||
**NO COVERAGE LOST.**
|
||||
|
||||
The cfold phase (`44e0242`) preserved the full pre-cfold custom-test set. All 64 custom tests are in canonical `tests/<recipe>/custom/` directories with per-recipe counts matching the pre-cfold baseline exactly. No assertions were weakened during the move. Deprecated `functional/`/`playwright/` aliases continue to discover and warn. Lifecycle overlays remain at top-level. The RUNG name `"functional"` is unchanged. The full real-CI sweep is green at L5 across all 20 enrolled recipes.
|
||||
|
||||
Reference in New Issue
Block a user