BACKLOG — server regression canaries phase

Build backlog

Create tests/regression/ suite (conftest + test_canaries + README)
Run good-simple canary (custom-html-tiny main) → confirm GREEN + test_serving passes
Run bad-false-green canary (custom-html v5-stale-docroot) → confirm RED + test_content_type fails
Run good-significant canary (lasuite-docs main) → confirm GREEN + test_serving_and_frontend passes
Open PR for operator review (DoD item 5: NOT merged)
Claim gate once all canary runs are GREEN/RED as expected + PR is open

Adversary findings

A-reg-1 [adversary] CLOSED @2026-06-02T01:46Z — relative import fixed, 3 tests collect

Filed: 2026-06-02T01:37Z Severity: CRITICAL — suite can't run at all until fixed

Cold-run cc-ci-run -m pytest tests/regression/ --collect-only on cc-ci confirms:

ImportError: attempted relative import with no known parent package
tests/regression/test_canaries.py:18: from .conftest import run_recipe_ci, ...

No tests collected. 0 canaries can run.

Root cause: test_canaries.py uses a relative import (from .conftest import ...) which requires the directory to be a Python package. Without tests/regression/__init__.py (and tests/__init__.py), pytest imports test_canaries.py as a top-level module, not a package member. Relative imports fail.

Repro:

ssh cc-ci
cd /root/builder-clone
cc-ci-run -m pytest tests/regression/ --collect-only
# → ImportError: attempted relative import with no known parent package

Fix (either approach):

Add tests/__init__.py and tests/regression/__init__.py (makes it a real package)
OR replace from .conftest import ... with absolute sys.path manipulation (like other test files do, e.g. sys.path.insert(0, ...); import conftest)

Adversary closes: after re-running --collect-only confirms 3+ tests collected, no error.

A-reg-3 [adversary] OPEN — CRITICAL: bad-backup and bad-restore fixtures broken (empty compose.yml)

Filed: 2026-06-02T01:58Z Severity: CRITICAL — both fixtures fail at upgrade instead of their intended tier

Cold-verified by inspecting regression-bad-backup and regression-bad-restore branches:

ssh cc-ci 'cd /root/.abra/recipes/custom-html && git diff origin/main..origin/regression-bad-backup -- compose.yml'

Result: compose.yml is completely empty (entire file deleted, leaving only a blank line). Same for regression-bad-restore.

Evidence from run artifacts:

regression-bad-backup-1: results: install=pass, upgrade=fail, backup=skip
- Expected: install=pass, upgrade=pass, backup=fail
- Actual: upgrade fails because chaos deploy deploys empty compose → no service → deploy error
regression-bad-restore-*: never ran to completion (same root cause blocks it)

Impact on regression test assertions: _assert_red_at_tier for bad-backup:

failing_tier="backup" → checks results["backup"]="skip" → FAIL: "expected 'backup'='fail', got 'skip'"
Test would FAIL with confusing assertion, not passing as expected

Fix: Recreate both fixture branches with correct compose.yml that:

bad-backup: keeps full valid nginx service, only changes backupbot.backup.path label to /nonexistent-cc-ci-canary-bad
bad-restore: keeps full valid nginx service, changes backup scope to capture a subdir that doesn't contain ci-marker.txt (so restore doesn't recover the marker)

The compose.yml should be identical to main EXCEPT for the single label/config change.

Repro: git diff origin/main..origin/regression-bad-backup -- compose.yml → empty file

Adversary closes: after both fixtures are recreated correctly, runs confirm:

bad-backup: install=pass, upgrade=pass, backup=fail
bad-restore: install=pass, upgrade=pass, backup=pass, restore=fail with test_restore_returns_state FAIL

A-reg-2 [adversary] OPEN — Plan gap: 4 per-tier RED canaries required by updated DoD

Filed: 2026-06-02T01:37Z Severity: HIGH — DoD#4 unmet; Builder cannot claim DONE without these

Updated plan (commit 7bdeb74) added DoD#4: four per-tier RED canaries (install/upgrade/backup/ restore on custom-html-tiny) that prove the server reports RED at EACH tier. Each must:

Assert overall verdict RED at the intended tier
Assert prior tiers PASSED
Have teeth: wrongly-green tier would FAIL the test

Current suite only has 3 canaries (good-simple, good-significant, bad-false-green). The 4 per-tier RED canaries are MISSING. This is a mandatory DoD item.

These also require:

Fixture branches or SHA-pinned commits where custom-html-tiny is broken at exactly one tier
A @pytest.mark.canary_fast sub-marker (plan recommends it for the fast RED subset)
README update to document the fast subset

Adversary closes: after all 4 canaries exist, run, and the Adversary cold-verifies each produces RED at the intended tier with prior tiers PASS.

4.9 KiB Raw Blame History