Files

autonomic-bot fd3db37c49

continuous-integration/drone/push Build is failing

Details

feat(regression): add tests/regression/ E2E canary suite

Three canaries (@pytest.mark.canary) drive the real cold CI lifecycle:
- good-simple: custom-html-tiny @ main (435df8fc) — fast signal, expects GREEN
- good-significant: lasuite-docs @ main (290a8ad7) — multi-service, expects GREEN
- bad-false-green: custom-html @ v5-stale-docroot (71e7326a) — expects RED

Semantic teeth: beyond exit-code, each test asserts that specific named tests
ran in results.json stages (test_serving, test_serving_and_frontend, test_content_type).
If an assertion is removed, the named test disappears → regression test fails.

Includes conftest (run_recipe_ci helper + stage_has_{passing,failing}_test),
README (cadence policy, how to run, how to add), and phase state files.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

2026-06-02 01:25:55 +00:00

3.1 KiB

Raw Blame History

STATUS — server regression canaries phase

Phase: server regression canaries (codified E2E self-tests) SSOT: /srv/cc-ci/cc-ci-plan/plan-server-regression-canaries.md Builder loop started: 2026-06-02 Repo: git.autonomic.zone/recipe-maintainers/cc-ci

Current state

Gate: D-initial CLAIMED — test suite written; awaiting first canary run

The tests/regression/ suite is committed. Before claiming the final gate (all DoD items verified), the canaries need to actually run on the live server and return the expected verdicts. Currently running the good-simple (custom-html-tiny) canary to confirm GREEN.

What was built

tests/regression/ committed in the cc-ci repo:

conftest.py — run_recipe_ci() helper that invokes the real harness as subprocess, returns (rc, results_dict, artifact_dir); stage_has_passing_test() / stage_has_failing_test() helpers for semantic checks
test_canaries.py — parametrized @pytest.mark.canary test with three canaries (see below)
README.md — cadence policy, how to run, how to add a canary

Canaries defined

ID	Recipe	SHA pinned	Expected
`good-simple`	`custom-html-tiny`	`435df8fc` (main 2026-06-02)	GREEN
`good-significant`	`lasuite-docs`	`290a8ad7` (main 2026-06-02)	GREEN
`bad-false-green`	`custom-html`	`71e7326a` (v5-stale-docroot)	RED

Semantic assertions (teeth)

Good canaries:

rc == 0 (harness exit)
install tier: "pass"
No tier is "fail"
flags.clean_teardown == True
flags.no_secret_leak == True
Named test test_serving present + passing in install stage (custom-html-tiny)
Named test test_serving_and_frontend present + passing in install stage (lasuite-docs)

Bad canary:

rc != 0 (PRIMARY — false-green catches here)
Named test test_content_type present + FAILING in custom stage (proves guard not vacuated)

How to verify (Adversary commands)

From cc-ci server root (requires the repo checked out at /root/cc-ci or similar):

# Good simple (fast ~2-5 min):
cc-ci-run python -m pytest tests/regression/ -m canary -k good-simple -v

# Bad canary (fast ~2-5 min, same recipe lifecycle):
cc-ci-run python -m pytest tests/regression/ -m canary -k bad-false-green -v

# Full suite (slow — lasuite-docs is 10-20 min):
cc-ci-run python -m pytest tests/regression/ -m canary -v

Expected outcomes:

good-simple: test PASSES (harness returns GREEN, test_serving passes)
bad-false-green: test PASSES (harness returns RED, test_content_type fails in custom stage)
good-significant: test PASSES (harness returns GREEN, test_serving_and_frontend passes)

Verify teeth: tamper with an outcome to confirm the regression test fails:

For good canary: unset test_serving (remove it) → stage_has_passing_test returns False → test fails
For bad canary: change the assert to rc == 0 → would fail if harness returns non-zero (teeth work)

In-flight

Canary good-simple running on cc-ci (started ~now)
Status will be updated once run completes and we have actual output to paste

3.1 KiB Raw Blame History