recipe-maintainers/cc-ci
0
0
Fork 0
Code Issues 1 Pull Requests 3 Actions Packages Projects Releases Wiki Activity
Files
a2a6eea75723f73175a850e929c8aac17ca8e637
cc-ci/machine-docs/BACKLOG-regression.md
autonomic-bot 464760ebb7
Some checks failed
continuous-integration/drone/push Build is failing
Details
review(regression): D-initial FAIL — A-reg-1 relative import (suite won't collect), A-reg-2 plan gap (4 per-tier RED canaries missing)
2026-06-02 01:34:56 +00:00

2.9 KiB
Raw Blame History

BACKLOG — server regression canaries phase

Build backlog

  • Create tests/regression/ suite (conftest + test_canaries + README)
  • Run good-simple canary (custom-html-tiny main) → confirm GREEN + test_serving passes
  • Run bad-false-green canary (custom-html v5-stale-docroot) → confirm RED + test_content_type fails
  • Run good-significant canary (lasuite-docs main) → confirm GREEN + test_serving_and_frontend passes
  • Open PR for operator review (DoD item 5: NOT merged)
  • Claim gate once all canary runs are GREEN/RED as expected + PR is open

Adversary findings

A-reg-1 [adversary] OPEN — CRITICAL: relative import fails, suite won't collect

Filed: 2026-06-02T01:37Z Severity: CRITICAL — suite can't run at all until fixed

Cold-run cc-ci-run -m pytest tests/regression/ --collect-only on cc-ci confirms:

ImportError: attempted relative import with no known parent package
tests/regression/test_canaries.py:18: from .conftest import run_recipe_ci, ...

No tests collected. 0 canaries can run.

Root cause: test_canaries.py uses a relative import (from .conftest import ...) which requires the directory to be a Python package. Without tests/regression/__init__.py (and tests/__init__.py), pytest imports test_canaries.py as a top-level module, not a package member. Relative imports fail.

Repro:

ssh cc-ci
cd /root/builder-clone
cc-ci-run -m pytest tests/regression/ --collect-only
# → ImportError: attempted relative import with no known parent package

Fix (either approach):

  1. Add tests/__init__.py and tests/regression/__init__.py (makes it a real package)
  2. OR replace from .conftest import ... with absolute sys.path manipulation (like other test files do, e.g. sys.path.insert(0, ...); import conftest)

Adversary closes: after re-running --collect-only confirms 3+ tests collected, no error.

A-reg-2 [adversary] OPEN — Plan gap: 4 per-tier RED canaries required by updated DoD

Filed: 2026-06-02T01:37Z Severity: HIGH — DoD#4 unmet; Builder cannot claim DONE without these

Updated plan (commit 7bdeb74) added DoD#4: four per-tier RED canaries (install/upgrade/backup/ restore on custom-html-tiny) that prove the server reports RED at EACH tier. Each must:

  • Assert overall verdict RED at the intended tier
  • Assert prior tiers PASSED
  • Have teeth: wrongly-green tier would FAIL the test

Current suite only has 3 canaries (good-simple, good-significant, bad-false-green). The 4 per-tier RED canaries are MISSING. This is a mandatory DoD item.

These also require:

  • Fixture branches or SHA-pinned commits where custom-html-tiny is broken at exactly one tier
  • A @pytest.mark.canary_fast sub-marker (plan recommends it for the fast RED subset)
  • README update to document the fast subset

Adversary closes: after all 4 canaries exist, run, and the Adversary cold-verifies each produces RED at the intended tier with prior tiers PASS.