cc-ci/machine-docs/BACKLOG-regression.md

BACKLOG — server regression canaries phase

Build backlog

• Create tests/regression/ suite (conftest + test_canaries + README)
• Run good-simple canary (custom-html-tiny main) → confirm GREEN + test_serving passes
• Run bad-false-green canary (custom-html v5-stale-docroot) → confirm RED + test_content_type fails
• Run good-significant canary (lasuite-docs main) → confirm GREEN + test_serving_and_frontend passes
• Open PR for operator review (DoD item 5: NOT merged)
• Claim gate once all canary runs are GREEN/RED as expected + PR is open

Adversary findings

A-reg-1 [adversary] CLOSED @2026-06-02T01:46Z — relative import fixed, 3 tests collect

Filed: 2026-06-02T01:37Z Severity: CRITICAL — suite can't run at all until fixed

Cold-run cc-ci-run -m pytest tests/regression/ --collect-only on cc-ci confirms:

ImportError: attempted relative import with no known parent package
tests/regression/test_canaries.py:18: from .conftest import run_recipe_ci, ...

No tests collected. 0 canaries can run.

Root cause: test_canaries.py uses a relative import (from .conftest import ...) which requires the directory to be a Python package. Without tests/regression/__init__.py (and tests/__init__.py), pytest imports test_canaries.py as a top-level module, not a package member. Relative imports fail.

Repro:

ssh cc-ci
cd /root/builder-clone
cc-ci-run -m pytest tests/regression/ --collect-only
# → ImportError: attempted relative import with no known parent package

Fix (either approach):

1. Add tests/__init__.py and tests/regression/__init__.py (makes it a real package)
2. OR replace from .conftest import ... with absolute sys.path manipulation (like other test files do, e.g. sys.path.insert(0, ...); import conftest)

Adversary closes: after re-running --collect-only confirms 3+ tests collected, no error.

---

A-reg-2 [adversary] OPEN — Plan gap: 4 per-tier RED canaries required by updated DoD

Filed: 2026-06-02T01:37Z Severity: HIGH — DoD#4 unmet; Builder cannot claim DONE without these

Updated plan (commit 7bdeb74) added DoD#4: four per-tier RED canaries (install/upgrade/backup/ restore on custom-html-tiny) that prove the server reports RED at EACH tier. Each must:

• Assert overall verdict RED at the intended tier
• Assert prior tiers PASSED
• Have teeth: wrongly-green tier would FAIL the test

Current suite only has 3 canaries (good-simple, good-significant, bad-false-green). The 4 per-tier RED canaries are MISSING. This is a mandatory DoD item.

These also require:

• Fixture branches or SHA-pinned commits where custom-html-tiny is broken at exactly one tier
• A @pytest.mark.canary_fast sub-marker (plan recommends it for the fast RED subset)
• README update to document the fast subset

Adversary closes: after all 4 canaries exist, run, and the Adversary cold-verifies each produces RED at the intended tier with prior tiers PASS.