121 lines
4.9 KiB
Markdown
121 lines
4.9 KiB
Markdown
# STATUS — server regression canaries phase
|
|
|
|
**Phase:** server regression canaries (codified E2E self-tests)
|
|
**SSOT:** `/srv/cc-ci/cc-ci-plan/plan-server-regression-canaries.md`
|
|
**Builder loop started:** 2026-06-02
|
|
**Repo:** git.autonomic.zone/recipe-maintainers/cc-ci
|
|
|
|
---
|
|
|
|
## Current state
|
|
|
|
**Gate: D-initial CLAIMED — test suite written; awaiting first canary run**
|
|
|
|
The `tests/regression/` suite is committed. Before claiming the final gate (all DoD items
|
|
verified), the canaries need to actually run on the live server and return the expected verdicts.
|
|
Currently running the good-simple (custom-html-tiny) canary to confirm GREEN.
|
|
|
|
---
|
|
|
|
## What was built
|
|
|
|
`tests/regression/` committed in the cc-ci repo:
|
|
- `conftest.py` — `run_recipe_ci()` helper that invokes the real harness as subprocess, returns `(rc, results_dict, artifact_dir)`; `stage_has_passing_test()` / `stage_has_failing_test()` helpers for semantic checks
|
|
- `test_canaries.py` — parametrized `@pytest.mark.canary` test with three canaries (see below)
|
|
- `README.md` — cadence policy, how to run, how to add a canary
|
|
|
|
---
|
|
|
|
## Canaries defined
|
|
|
|
| ID | Recipe | SHA pinned | Expected |
|
|
|----|--------|-----------|----------|
|
|
| `good-simple` | `custom-html-tiny` | `435df8fc` (main 2026-06-02) | GREEN |
|
|
| `good-significant` | `lasuite-docs` | `290a8ad7` (main 2026-06-02) | GREEN |
|
|
| `bad-false-green` | `custom-html` | `71e7326a` (v5-stale-docroot) | RED |
|
|
|
|
---
|
|
|
|
## Semantic assertions (teeth)
|
|
|
|
Good canaries:
|
|
- `rc == 0` (harness exit)
|
|
- install tier: "pass"
|
|
- No tier is "fail"
|
|
- `flags.clean_teardown == True`
|
|
- `flags.no_secret_leak == True`
|
|
- Named test `test_serving` present + passing in install stage (custom-html-tiny)
|
|
- Named test `test_serving_and_frontend` present + passing in install stage (lasuite-docs)
|
|
|
|
Bad canary:
|
|
- `rc != 0` (PRIMARY — false-green catches here)
|
|
- Named test `test_content_type` present + FAILING in custom stage (proves guard not vacuated)
|
|
|
|
---
|
|
|
|
## How to verify (Adversary commands)
|
|
|
|
From cc-ci server root (requires the repo checked out at `/root/cc-ci` or similar):
|
|
|
|
```bash
|
|
# Good simple (fast ~2-5 min):
|
|
cc-ci-run python -m pytest tests/regression/ -m canary -k good-simple -v
|
|
|
|
# Bad canary (fast ~2-5 min, same recipe lifecycle):
|
|
cc-ci-run python -m pytest tests/regression/ -m canary -k bad-false-green -v
|
|
|
|
# Full suite (slow — lasuite-docs is 10-20 min):
|
|
cc-ci-run python -m pytest tests/regression/ -m canary -v
|
|
```
|
|
|
|
Expected outcomes:
|
|
- `good-simple`: test PASSES (harness returns GREEN, test_serving passes)
|
|
- `bad-false-green`: test PASSES (harness returns RED, test_content_type fails in custom stage)
|
|
- `good-significant`: test PASSES (harness returns GREEN, test_serving_and_frontend passes)
|
|
|
|
Verify teeth: tamper with an outcome to confirm the regression test fails:
|
|
- For good canary: unset `test_serving` (remove it) → `stage_has_passing_test` returns False → test fails
|
|
- For bad canary: change the assert to `rc == 0` → would fail if harness returns non-zero (teeth work)
|
|
|
|
---
|
|
|
|
## Canary run results (2026-06-02 ~01:28-01:35Z)
|
|
|
|
### bad-false-green ✓ (RED confirmed)
|
|
Run ID: `regression-bad-canary-1`, artifact: `/var/lib/cc-ci-runs/regression-bad-canary-1/`
|
|
```
|
|
results: install=pass, upgrade=pass, backup=pass, restore=pass, custom=FAIL
|
|
level: 3 (L4 functional FAILED)
|
|
flags: clean_teardown=True, no_secret_leak=True
|
|
stages.custom tests: [test_content_roundtrip, test_content_type_html_and_txt(FAIL), test_custom_html_returns_200, test_browser_renders_html]
|
|
rc: 1 (any(fail in results))
|
|
```
|
|
Confirms: `test_content_type_html_and_txt` fails with `Content-Type='application/octet-stream'`, expected `text/plain`. The regression test `assert rc != 0` PASSES.
|
|
|
|
### good-simple ✓ (GREEN confirmed)
|
|
Run ID: `regression-good-simple-1`, artifact: `/var/lib/cc-ci-runs/regression-good-simple-1/`
|
|
```
|
|
results: install=pass, upgrade=pass, backup=skip, restore=skip, custom=skip
|
|
level: 2 (L3 backup/restore N/A — no backupbot label)
|
|
flags: clean_teardown=True, no_secret_leak=True
|
|
stages.install tests: [test_serving (PASS)]
|
|
rc: 0
|
|
```
|
|
Confirms: `test_serving` present + passing in install stage. All assertions will pass.
|
|
|
|
### good-significant (FAILED upgrade — transient convergence race suspected)
|
|
Run ID: `regression-good-significant-1`, artifact: `/var/lib/cc-ci-runs/regression-good-significant-1/`
|
|
```
|
|
results: install=PASS, upgrade=FAIL, backup=pass, restore=pass, custom=pass
|
|
level: 1 (L2 upgrade FAILED)
|
|
```
|
|
Failure: `test_upgrade_reconverges` → `assert_serving` failed — 9-service stack didn't converge
|
|
within the assert window after chaos redeploy. This is the known WOPI convergence race.
|
|
TODO: re-run to confirm transient; adjust good-significant test if flaky.
|
|
|
|
### NEXT STEPS
|
|
1. Re-run good-significant (lasuite-docs) — confirm transient upgrade race
|
|
2. Create 4 per-tier RED canary branches on Gitea mirror (custom-html-tiny for install/upgrade, custom-html for backup/restore)
|
|
3. Add 4 RED canary tests to test_canaries.py
|
|
4. Commit + open PR
|