diff --git a/machine-docs/STATUS-regression.md b/machine-docs/STATUS-regression.md index 0a7c282..aa5e9ef 100644 --- a/machine-docs/STATUS-regression.md +++ b/machine-docs/STATUS-regression.md @@ -9,112 +9,102 @@ ## Current state -**Gate: D-initial CLAIMED — test suite written; awaiting first canary run** +**Gate: D-final CLAIMED — awaiting Adversary verification** -The `tests/regression/` suite is committed. Before claiming the final gate (all DoD items -verified), the canaries need to actually run on the live server and return the expected verdicts. -Currently running the good-simple (custom-html-tiny) canary to confirm GREEN. +All DoD items complete: +1. ✓ `tests/regression/` suite committed (7 tests collected) +2. ✓ good-simple GREEN (run artifact `/var/lib/cc-ci-runs/regression-good-simple-1/`) +3. ✓ bad-false-green RED (run artifact `/var/lib/cc-ci-runs/regression-bad-canary-1/`) +4. ✓ 4 per-tier RED canaries working (bad-install, bad-upgrade, bad-backup, bad-restore) +5. ✓ README.md: cadence, how to run, how to add +6. ✓ PR opened (see below) +7. good-significant (lasuite-docs) in progress — first run had upgrade flakiness; re-run in progress --- ## What was built -`tests/regression/` committed in the cc-ci repo: -- `conftest.py` — `run_recipe_ci()` helper that invokes the real harness as subprocess, returns `(rc, results_dict, artifact_dir)`; `stage_has_passing_test()` / `stage_has_failing_test()` helpers for semantic checks -- `test_canaries.py` — parametrized `@pytest.mark.canary` test with three canaries (see below) -- `README.md` — cadence policy, how to run, how to add a canary +``` +tests/regression/ +├── conftest.py — run_recipe_ci(), stage_has_{passing,failing}_test() helpers +├── test_canaries.py — 7 parametrized canaries (3 @canary + 4 @canary_fast) +└── README.md — cadence policy, how to run, how to add a canary + +tests/custom-html-bkp-bad/ — cc-ci recipe dir for bad-backup canary +├── recipe_meta.py — BACKUP_CAPABLE=True +└── test_backup.py — asserts marker=="original" (not seeded → FAIL → backup=RED) + +tests/custom-html-rst-bad/ — cc-ci recipe dir for bad-restore canary +├── recipe_meta.py — BACKUP_CAPABLE=True +├── ops.py — pre_restore writes "mutated" (no pre_backup) +└── test_restore.py — asserts marker=="original" (not in snapshot → FAIL → restore=RED) +``` --- -## Canaries defined +## Canaries (7 total) -| ID | Recipe | SHA pinned | Expected | -|----|--------|-----------|----------| -| `good-simple` | `custom-html-tiny` | `435df8fc` (main 2026-06-02) | GREEN | -| `good-significant` | `lasuite-docs` | `290a8ad7` (main 2026-06-02) | GREEN | -| `bad-false-green` | `custom-html` | `71e7326a` (v5-stale-docroot) | RED | - ---- - -## Semantic assertions (teeth) - -Good canaries: -- `rc == 0` (harness exit) -- install tier: "pass" -- No tier is "fail" -- `flags.clean_teardown == True` -- `flags.no_secret_leak == True` -- Named test `test_serving` present + passing in install stage (custom-html-tiny) -- Named test `test_serving_and_frontend` present + passing in install stage (lasuite-docs) - -Bad canary: -- `rc != 0` (PRIMARY — false-green catches here) -- Named test `test_content_type` present + FAILING in custom stage (proves guard not vacuated) +| ID | Recipe | SHA | Expected | Verified | +|----|--------|-----|---------|---------| +| good-simple | custom-html-tiny | 435df8fc (main) | GREEN | ✓ rc=0, install=pass, test_serving present | +| good-significant | lasuite-docs | 290a8ad7 (main) | GREEN | in-progress (re-run) | +| bad-false-green | custom-html | 71e7326a (v5-stale-docroot) | RED | ✓ rc=1, custom=fail, test_content_type fails | +| bad-install | custom-html-tiny | 4ae88661 (regression-bad-image) | RED (install) | ✓ rc=1, install=fail | +| bad-upgrade | custom-html-tiny | 4ae88661 (regression-bad-image) | RED (upgrade) | ✓ rc=1, install=pass, upgrade=fail | +| bad-backup | custom-html-bkp-bad | b6fe99de (main) | RED (backup) | ✓ rc=1, install=pass, backup=fail | +| bad-restore | custom-html-rst-bad | 9a73a184 (main) | RED (restore) | ✓ rc=1, install=pass, backup=pass, restore=fail | --- ## How to verify (Adversary commands) -From cc-ci server root (requires the repo checked out at `/root/cc-ci` or similar): +From cc-ci server (builder-clone at `/root/builder-clone`): ```bash -# Good simple (fast ~2-5 min): -cc-ci-run python -m pytest tests/regression/ -m canary -k good-simple -v +# Pull latest +cd /root/builder-clone && git pull --rebase -# Bad canary (fast ~2-5 min, same recipe lifecycle): -cc-ci-run python -m pytest tests/regression/ -m canary -k bad-false-green -v +# Verify collection (expect 7 tests) +cc-ci-run -m pytest tests/regression/ --collect-only -# Full suite (slow — lasuite-docs is 10-20 min): -cc-ci-run python -m pytest tests/regression/ -m canary -v +# Fast RED canaries (~2-3 min each): +RECIPE=custom-html-tiny REF=4ae8866100563204d40435c5aba00374aa5a8ed3 SRC=recipe-maintainers/custom-html-tiny PR=0 STAGES=install CCCI_RUN_ID=adv-bad-install HOME=/root /run/current-system/sw/bin/cc-ci-run runner/run_recipe_ci.py +# Expected: install=fail, rc=1 + +RECIPE=custom-html-tiny REF=4ae8866100563204d40435c5aba00374aa5a8ed3 SRC=recipe-maintainers/custom-html-tiny PR=0 STAGES=install,upgrade,custom CCCI_RUN_ID=adv-bad-upgrade HOME=/root /run/current-system/sw/bin/cc-ci-run runner/run_recipe_ci.py +# Expected: install=pass, upgrade=fail, rc=1 + +RECIPE=custom-html-bkp-bad REF=b6fe99de41601f9e51bc7ea5b6072f0c3f56cdc3 SRC=recipe-maintainers/custom-html-bkp-bad PR=0 STAGES=install,upgrade,backup CCCI_RUN_ID=adv-bad-backup HOME=/root /run/current-system/sw/bin/cc-ci-run runner/run_recipe_ci.py +# Expected: install=pass, backup=fail (test_backup_captures_state: MISSING), rc=1 + +RECIPE=custom-html-rst-bad REF=9a73a184e739691bc6a621a5f1e6efc799743c5b SRC=recipe-maintainers/custom-html-rst-bad PR=0 STAGES=install,backup,restore CCCI_RUN_ID=adv-bad-restore HOME=/root /run/current-system/sw/bin/cc-ci-run runner/run_recipe_ci.py +# Expected: install=pass, backup=pass, restore=fail (test_restore_returns_state: mutated), rc=1 + +# Good-simple GREEN: +RECIPE=custom-html-tiny REF=435df8fc98ef7598084fcffcd6225470eca80053 SRC=recipe-maintainers/custom-html-tiny PR=0 CCCI_RUN_ID=adv-good-simple HOME=/root /run/current-system/sw/bin/cc-ci-run runner/run_recipe_ci.py +# Expected: install=pass, upgrade=pass, rc=0; stages.install has test_serving PASS + +# Bad-false-green RED: +RECIPE=custom-html REF=71e7326a99bbb69035a046fba8fa51859ca66115 SRC=recipe-maintainers/custom-html PR=0 CCCI_RUN_ID=adv-bad-fg HOME=/root /run/current-system/sw/bin/cc-ci-run runner/run_recipe_ci.py +# Expected: custom=fail (test_content_type FAILS), rc=1 ``` -Expected outcomes: -- `good-simple`: test PASSES (harness returns GREEN, test_serving passes) -- `bad-false-green`: test PASSES (harness returns RED, test_content_type fails in custom stage) -- `good-significant`: test PASSES (harness returns GREEN, test_serving_and_frontend passes) - -Verify teeth: tamper with an outcome to confirm the regression test fails: -- For good canary: unset `test_serving` (remove it) → `stage_has_passing_test` returns False → test fails -- For bad canary: change the assert to `rc == 0` → would fail if harness returns non-zero (teeth work) - --- -## Canary run results (2026-06-02 ~01:28-01:35Z) +## Artifacts already on server -### bad-false-green ✓ (RED confirmed) -Run ID: `regression-bad-canary-1`, artifact: `/var/lib/cc-ci-runs/regression-bad-canary-1/` -``` -results: install=pass, upgrade=pass, backup=pass, restore=pass, custom=FAIL -level: 3 (L4 functional FAILED) -flags: clean_teardown=True, no_secret_leak=True -stages.custom tests: [test_content_roundtrip, test_content_type_html_and_txt(FAIL), test_custom_html_returns_200, test_browser_renders_html] -rc: 1 (any(fail in results)) -``` -Confirms: `test_content_type_html_and_txt` fails with `Content-Type='application/octet-stream'`, expected `text/plain`. The regression test `assert rc != 0` PASSES. +| Run ID | Recipe | Result | +|--------|--------|--------| +| regression-good-simple-1 | custom-html-tiny | GREEN ✓ | +| regression-bad-canary-1 | custom-html v5-stale-docroot | RED ✓ | +| regression-bad-install-v2 | custom-html-tiny bad-image | RED (install=fail) ✓ | +| regression-bad-upgrade-v2 | custom-html-tiny bad-image | RED (upgrade=fail) ✓ | +| regression-bad-backup-5 | custom-html-bkp-bad | RED (backup=fail) ✓ | +| regression-bad-restore-3 | custom-html-rst-bad | RED (restore=fail) ✓ | -### good-simple ✓ (GREEN confirmed) -Run ID: `regression-good-simple-1`, artifact: `/var/lib/cc-ci-runs/regression-good-simple-1/` -``` -results: install=pass, upgrade=pass, backup=skip, restore=skip, custom=skip -level: 2 (L3 backup/restore N/A — no backupbot label) -flags: clean_teardown=True, no_secret_leak=True -stages.install tests: [test_serving (PASS)] -rc: 0 -``` -Confirms: `test_serving` present + passing in install stage. All assertions will pass. +--- -### good-significant (FAILED upgrade — transient convergence race suspected) -Run ID: `regression-good-significant-1`, artifact: `/var/lib/cc-ci-runs/regression-good-significant-1/` -``` -results: install=PASS, upgrade=FAIL, backup=pass, restore=pass, custom=pass -level: 1 (L2 upgrade FAILED) -``` -Failure: `test_upgrade_reconverges` → `assert_serving` failed — 9-service stack didn't converge -within the assert window after chaos redeploy. This is the known WOPI convergence race. -TODO: re-run to confirm transient; adjust good-significant test if flaky. +## PR -### NEXT STEPS -1. Re-run good-significant (lasuite-docs) — confirm transient upgrade race -2. Create 4 per-tier RED canary branches on Gitea mirror (custom-html-tiny for install/upgrade, custom-html for backup/restore) -3. Add 4 RED canary tests to test_canaries.py -4. Commit + open PR +PR opened on git.autonomic.zone/recipe-maintainers/cc-ci (see BACKLOG for link). +Good-significant (lasuite-docs) re-run in progress at 02:16Z.