132 lines
6.6 KiB
Markdown
132 lines
6.6 KiB
Markdown
# BACKLOG — server regression canaries phase
|
|
|
|
## Build backlog
|
|
|
|
- [x] Create `tests/regression/` suite (conftest + test_canaries + README)
|
|
- [ ] Run `good-simple` canary (custom-html-tiny main) → confirm GREEN + test_serving passes
|
|
- [ ] Run `bad-false-green` canary (custom-html v5-stale-docroot) → confirm RED + test_content_type fails
|
|
- [ ] Run `good-significant` canary (lasuite-docs main) → confirm GREEN + test_serving_and_frontend passes
|
|
- [ ] Open PR for operator review (DoD item 5: NOT merged)
|
|
- [ ] Claim gate once all canary runs are GREEN/RED as expected + PR is open
|
|
|
|
## Adversary findings
|
|
|
|
### A-reg-1 [adversary] CLOSED @2026-06-02T01:46Z — relative import fixed, 3 tests collect
|
|
**Filed:** 2026-06-02T01:37Z
|
|
**Severity:** CRITICAL — suite can't run at all until fixed
|
|
|
|
Cold-run `cc-ci-run -m pytest tests/regression/ --collect-only` on cc-ci confirms:
|
|
```
|
|
ImportError: attempted relative import with no known parent package
|
|
tests/regression/test_canaries.py:18: from .conftest import run_recipe_ci, ...
|
|
```
|
|
No tests collected. 0 canaries can run.
|
|
|
|
**Root cause:** `test_canaries.py` uses a relative import (`from .conftest import ...`) which
|
|
requires the directory to be a Python package. Without `tests/regression/__init__.py` (and
|
|
`tests/__init__.py`), pytest imports `test_canaries.py` as a top-level module, not a package
|
|
member. Relative imports fail.
|
|
|
|
**Repro:**
|
|
```bash
|
|
ssh cc-ci
|
|
cd /root/builder-clone
|
|
cc-ci-run -m pytest tests/regression/ --collect-only
|
|
# → ImportError: attempted relative import with no known parent package
|
|
```
|
|
|
|
**Fix (either approach):**
|
|
1. Add `tests/__init__.py` and `tests/regression/__init__.py` (makes it a real package)
|
|
2. OR replace `from .conftest import ...` with absolute sys.path manipulation (like other test
|
|
files do, e.g. `sys.path.insert(0, ...); import conftest`)
|
|
|
|
**Adversary closes:** after re-running `--collect-only` confirms 3+ tests collected, no error.
|
|
|
|
---
|
|
|
|
### A-reg-3 [adversary] CLOSED @2026-06-02T02:20Z — fixtures fixed; cold-verified correct tier failures
|
|
|
|
**Resolved:** Builder created separate recipes (`custom-html-bkp-bad`, `custom-html-rst-bad`) with
|
|
correct fixture structure. Cold-verified from cc-ci artifact dirs (no harness re-run needed).
|
|
|
|
**Evidence:**
|
|
- bad-backup-5 (`b6fe99de`, custom-html-bkp-bad): `install=pass, backup=fail` ✓
|
|
- `test_backup_artifact: pass` (snapshot IS produced)
|
|
- `test_backup_captures_state: fail` ("MISSING" not "original") ✓ — backup=RED
|
|
- bad-restore-3 (`9a73a184e739`, custom-html-rst-bad): `install=pass, backup=pass, restore=fail` ✓
|
|
- `test_restore_returns_state: fail` ("mutated" not "original") ✓ — restore=RED
|
|
|
|
### A-reg-3 [adversary] OPEN — CRITICAL: bad-backup and bad-restore fixtures broken (empty compose.yml)
|
|
**Filed:** 2026-06-02T01:58Z
|
|
**Severity:** CRITICAL — both fixtures fail at upgrade instead of their intended tier
|
|
|
|
Cold-verified by inspecting `regression-bad-backup` and `regression-bad-restore` branches:
|
|
```bash
|
|
ssh cc-ci 'cd /root/.abra/recipes/custom-html && git diff origin/main..origin/regression-bad-backup -- compose.yml'
|
|
```
|
|
Result: compose.yml is completely empty (entire file deleted, leaving only a blank line). Same
|
|
for `regression-bad-restore`.
|
|
|
|
**Evidence from run artifacts:**
|
|
- `regression-bad-backup-1`: `results: install=pass, upgrade=fail, backup=skip`
|
|
- Expected: `install=pass, upgrade=pass, backup=fail`
|
|
- Actual: upgrade fails because chaos deploy deploys empty compose → no service → deploy error
|
|
- `regression-bad-restore-*`: never ran to completion (same root cause blocks it)
|
|
|
|
**Impact on regression test assertions:**
|
|
`_assert_red_at_tier` for bad-backup:
|
|
- `failing_tier="backup"` → checks `results["backup"]="skip"` → FAIL: "expected 'backup'='fail', got 'skip'"
|
|
- Test would FAIL with confusing assertion, not passing as expected
|
|
|
|
**Fix:** Recreate both fixture branches with correct compose.yml that:
|
|
- bad-backup: keeps full valid nginx service, only changes `backupbot.backup.path` label to `/nonexistent-cc-ci-canary-bad`
|
|
- bad-restore: keeps full valid nginx service, changes backup scope to capture a subdir that doesn't contain ci-marker.txt (so restore doesn't recover the marker)
|
|
|
|
The compose.yml should be identical to main EXCEPT for the single label/config change.
|
|
|
|
**Repro:** `git diff origin/main..origin/regression-bad-backup -- compose.yml` → empty file
|
|
|
|
**Adversary closes:** after both fixtures are recreated correctly, runs confirm:
|
|
- bad-backup: `install=pass, upgrade=pass, backup=fail`
|
|
- bad-restore: `install=pass, upgrade=pass, backup=pass, restore=fail` with `test_restore_returns_state` FAIL
|
|
|
|
---
|
|
|
|
### A-reg-2 [adversary] CLOSED @2026-06-02T02:20Z — 4 per-tier RED canaries cold-verified
|
|
|
|
**Resolved:** All 4 per-tier RED canaries added, artifacts cold-verified on cc-ci.
|
|
|
|
| Canary | Run artifact | failing_tier | passing_before | verdict |
|
|
|--------|-------------|-------------|---------------|---------|
|
|
| bad-install | regression-bad-install-v2 | install=fail ✓ | [] | CORRECT ✓ |
|
|
| bad-upgrade | regression-bad-upgrade-v2 | upgrade=fail ✓ | install=pass ✓ | CORRECT ✓ |
|
|
| bad-backup | regression-bad-backup-5 | backup=fail ✓ | install=pass ✓ | CORRECT ✓ |
|
|
| bad-restore | regression-bad-restore-3 | restore=fail ✓ | install=pass, backup=pass ✓ | CORRECT ✓ |
|
|
|
|
`@pytest.mark.canary_fast` marker added ✓. 7 tests collect ✓.
|
|
|
|
**Note:** bad-backup comment in test_canaries.py says "test_backup_artifact fails" but actual
|
|
behavior is test_backup_artifact PASSES and test_backup_captures_state FAILS. Functional result
|
|
(backup=fail) is correct; comment is misleading but non-blocking.
|
|
|
|
### A-reg-2 [adversary] OPEN — Plan gap: 4 per-tier RED canaries required by updated DoD
|
|
**Filed:** 2026-06-02T01:37Z
|
|
**Severity:** HIGH — DoD#4 unmet; Builder cannot claim DONE without these
|
|
|
|
Updated plan (commit 7bdeb74) added DoD#4: four per-tier RED canaries (install/upgrade/backup/
|
|
restore on `custom-html-tiny`) that prove the server reports RED at EACH tier. Each must:
|
|
- Assert overall verdict RED at the intended tier
|
|
- Assert prior tiers PASSED
|
|
- Have teeth: wrongly-green tier would FAIL the test
|
|
|
|
Current suite only has 3 canaries (good-simple, good-significant, bad-false-green). The 4
|
|
per-tier RED canaries are MISSING. This is a mandatory DoD item.
|
|
|
|
These also require:
|
|
- Fixture branches or SHA-pinned commits where custom-html-tiny is broken at exactly one tier
|
|
- A `@pytest.mark.canary_fast` sub-marker (plan recommends it for the fast RED subset)
|
|
- README update to document the fast subset
|
|
|
|
**Adversary closes:** after all 4 canaries exist, run, and the Adversary cold-verifies each
|
|
produces RED at the intended tier with prior tiers PASS.
|