cc-ci/machine-docs/BACKLOG-regression.md

BACKLOG — server regression canaries phase

Build backlog

• Create tests/regression/ suite (conftest + test_canaries + README)
• Run good-simple canary (custom-html-tiny main) → confirm GREEN + test_serving passes
• Run bad-false-green canary (custom-html v5-stale-docroot) → confirm RED + test_content_type fails
• Run good-significant canary (lasuite-docs main) → confirm GREEN + test_serving_and_frontend passes
• Open PR for operator review (DoD item 5: NOT merged)
• Claim gate once all canary runs are GREEN/RED as expected + PR is open

Adversary findings

A-reg-1 [adversary] CLOSED @2026-06-02T01:46Z — relative import fixed, 3 tests collect

Filed: 2026-06-02T01:37Z Severity: CRITICAL — suite can't run at all until fixed

Cold-run cc-ci-run -m pytest tests/regression/ --collect-only on cc-ci confirms:

ImportError: attempted relative import with no known parent package
tests/regression/test_canaries.py:18: from .conftest import run_recipe_ci, ...

No tests collected. 0 canaries can run.

Root cause: test_canaries.py uses a relative import (from .conftest import ...) which requires the directory to be a Python package. Without tests/regression/__init__.py (and tests/__init__.py), pytest imports test_canaries.py as a top-level module, not a package member. Relative imports fail.

Repro:

ssh cc-ci
cd /root/builder-clone
cc-ci-run -m pytest tests/regression/ --collect-only
# → ImportError: attempted relative import with no known parent package

Fix (either approach):

1. Add tests/__init__.py and tests/regression/__init__.py (makes it a real package)
2. OR replace from .conftest import ... with absolute sys.path manipulation (like other test files do, e.g. sys.path.insert(0, ...); import conftest)

Adversary closes: after re-running --collect-only confirms 3+ tests collected, no error.

---

A-reg-3 [adversary] CLOSED @2026-06-02T02:20Z — fixtures fixed; cold-verified correct tier failures

Resolved: Builder created separate recipes (custom-html-bkp-bad, custom-html-rst-bad) with correct fixture structure. Cold-verified from cc-ci artifact dirs (no harness re-run needed).

Evidence:

• bad-backup-5 (b6fe99de, custom-html-bkp-bad): install=pass, backup=fail ✓
  • test_backup_artifact: pass (snapshot IS produced)
  • test_backup_captures_state: fail ("MISSING" not "original") ✓ — backup=RED
• bad-restore-3 (9a73a184e739, custom-html-rst-bad): install=pass, backup=pass, restore=fail ✓
  • test_restore_returns_state: fail ("mutated" not "original") ✓ — restore=RED

A-reg-3 [adversary] OPEN — CRITICAL: bad-backup and bad-restore fixtures broken (empty compose.yml)

Filed: 2026-06-02T01:58Z Severity: CRITICAL — both fixtures fail at upgrade instead of their intended tier

Cold-verified by inspecting regression-bad-backup and regression-bad-restore branches:

ssh cc-ci 'cd /root/.abra/recipes/custom-html && git diff origin/main..origin/regression-bad-backup -- compose.yml'

Result: compose.yml is completely empty (entire file deleted, leaving only a blank line). Same for regression-bad-restore.

Evidence from run artifacts:

• regression-bad-backup-1: results: install=pass, upgrade=fail, backup=skip
  • Expected: install=pass, upgrade=pass, backup=fail
  • Actual: upgrade fails because chaos deploy deploys empty compose → no service → deploy error
• regression-bad-restore-*: never ran to completion (same root cause blocks it)

Impact on regression test assertions: _assert_red_at_tier for bad-backup:

• failing_tier="backup" → checks results["backup"]="skip" → FAIL: "expected 'backup'='fail', got 'skip'"
• Test would FAIL with confusing assertion, not passing as expected

Fix: Recreate both fixture branches with correct compose.yml that:

• bad-backup: keeps full valid nginx service, only changes backupbot.backup.path label to /nonexistent-cc-ci-canary-bad
• bad-restore: keeps full valid nginx service, changes backup scope to capture a subdir that doesn't contain ci-marker.txt (so restore doesn't recover the marker)

The compose.yml should be identical to main EXCEPT for the single label/config change.

Repro: git diff origin/main..origin/regression-bad-backup -- compose.yml → empty file

Adversary closes: after both fixtures are recreated correctly, runs confirm:

• bad-backup: install=pass, upgrade=pass, backup=fail
• bad-restore: install=pass, upgrade=pass, backup=pass, restore=fail with test_restore_returns_state FAIL

---

A-reg-2 [adversary] CLOSED @2026-06-02T02:20Z — 4 per-tier RED canaries cold-verified

Resolved: All 4 per-tier RED canaries added, artifacts cold-verified on cc-ci.

Canary	Run artifact	failing_tier	passing_before	verdict
bad-install	regression-bad-install-v2	install=fail ✓	[]	CORRECT ✓
bad-upgrade	regression-bad-upgrade-v2	upgrade=fail ✓	install=pass ✓	CORRECT ✓
bad-backup	regression-bad-backup-5	backup=fail ✓	install=pass ✓	CORRECT ✓
bad-restore	regression-bad-restore-3	restore=fail ✓	install=pass, backup=pass ✓	CORRECT ✓

@pytest.mark.canary_fast marker added ✓. 7 tests collect ✓.

Note: bad-backup comment in test_canaries.py says "test_backup_artifact fails" but actual behavior is test_backup_artifact PASSES and test_backup_captures_state FAILS. Functional result (backup=fail) is correct; comment is misleading but non-blocking.

A-reg-2 [adversary] OPEN — Plan gap: 4 per-tier RED canaries required by updated DoD

Filed: 2026-06-02T01:37Z Severity: HIGH — DoD#4 unmet; Builder cannot claim DONE without these

Updated plan (commit 7bdeb74) added DoD#4: four per-tier RED canaries (install/upgrade/backup/ restore on custom-html-tiny) that prove the server reports RED at EACH tier. Each must:

• Assert overall verdict RED at the intended tier
• Assert prior tiers PASSED
• Have teeth: wrongly-green tier would FAIL the test

Current suite only has 3 canaries (good-simple, good-significant, bad-false-green). The 4 per-tier RED canaries are MISSING. This is a mandatory DoD item.

These also require:

• Fixture branches or SHA-pinned commits where custom-html-tiny is broken at exactly one tier
• A @pytest.mark.canary_fast sub-marker (plan recommends it for the fast RED subset)
• README update to document the fast subset

Adversary closes: after all 4 canaries exist, run, and the Adversary cold-verifies each produces RED at the intended tier with prior tiers PASS.