recipe-maintainers/cc-ci
0
0
Fork 0
Code Issues 1 Pull Requests 3 Actions Packages Projects Releases Wiki Activity
Files
fd3db37c49dcbbb8fd31155b7d79e95bac96e4f2
cc-ci/machine-docs/JOURNAL-regression.md
autonomic-bot fd3db37c49
Some checks failed
continuous-integration/drone/push Build is failing
Details
feat(regression): add tests/regression/ E2E canary suite 
Three canaries (@pytest.mark.canary) drive the real cold CI lifecycle:
- good-simple: custom-html-tiny @ main (435df8fc) — fast signal, expects GREEN
- good-significant: lasuite-docs @ main (290a8ad7) — multi-service, expects GREEN
- bad-false-green: custom-html @ v5-stale-docroot (71e7326a) — expects RED

Semantic teeth: beyond exit-code, each test asserts that specific named tests
ran in results.json stages (test_serving, test_serving_and_frontend, test_content_type).
If an assertion is removed, the named test disappears → regression test fails.

Includes conftest (run_recipe_ci helper + stage_has_{passing,failing}_test),
README (cadence policy, how to run, how to add), and phase state files.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-02 01:25:55 +00:00

2.4 KiB
Raw Blame History

JOURNAL — server regression canaries phase (Builder)

Phase: server regression canaries Started: 2026-06-02

Step 0 — phase kickoff and design (2026-06-02)

Context: Mirror phase (plan-mirror-enroll-all-recipes.md) completed DONE at 2026-06-02T01:16Z. Adversary initialized regression phase files in machine-docs/ at commit f202c5a.

Decision: run regression tests ON cc-ci, not from the orchestrator

The regression tests call run_recipe_ci.py which uses abra/docker/swarm — these only exist on cc-ci. The test process runs under cc-ci-run python -m pytest, which sets up the right PATH (abra, python3, playwright, etc.). The test then invokes run_recipe_ci.py as a subprocess using sys.executable (inherits the same python3 from cc-ci-run).

The README.md documents the ssh cc-ci "cc-ci-run python -m pytest tests/regression/ -m canary" invocation pattern.

Canary selection:

ID Recipe SHA Rationale
good-simple custom-html-tiny 435df8fc (main) Fast, few deps, quick signal
good-significant lasuite-docs 290a8ad7 (main) Multi-service, exercises real breadth
bad-false-green custom-html 71e7326a (v5-stale-docroot) Already produced RED build #75; pinned fixture

SHAs confirmed from Gitea API on 2026-06-02.

Semantic checks ("teeth") design:

The regression tests assert BOTH exit code AND named tests in results.json stages. This guards against two failure modes:

  1. Harness returns wrong exit code (false-green / false-red) → rc assertion catches it
  2. A specific assertion is silently removed/vacuated → named test disappears from stages → semantic check catches it

For custom-html-tiny: test_serving (generic install) must appear passing For lasuite-docs: test_serving_and_frontend (install overlay) must appear passing For bad canary: test_content_type (custom functional) must appear failing

File layout:

  • tests/regression/conftest.py — run_recipe_ci(), stage_has_passing_test(), stage_has_failing_test()
  • tests/regression/test_canaries.py — parametrized @pytest.mark.canary test
  • tests/regression/README.md — cadence policy + how to run + how to add

Next step: commit + push, then run good-simple and bad-false-green canaries to get real output. lasuite-docs is slow (10-20 min) so will run it last.

Step 1 — initial canary runs (in progress, 2026-06-02)

Committed suite, will record canary outputs here as they complete.