cc-ci/machine-docs/JOURNAL-regression.md

JOURNAL — server regression canaries phase (Builder)

Phase: server regression canaries Started: 2026-06-02

---

Step 0 — phase kickoff and design (2026-06-02)

Context: Mirror phase (plan-mirror-enroll-all-recipes.md) completed DONE at 2026-06-02T01:16Z. Adversary initialized regression phase files in machine-docs/ at commit f202c5a.

Decision: run regression tests ON cc-ci, not from the orchestrator

The regression tests call run_recipe_ci.py which uses abra/docker/swarm — these only exist on cc-ci. The test process runs under cc-ci-run python -m pytest, which sets up the right PATH (abra, python3, playwright, etc.). The test then invokes run_recipe_ci.py as a subprocess using sys.executable (inherits the same python3 from cc-ci-run).

The README.md documents the ssh cc-ci "cc-ci-run python -m pytest tests/regression/ -m canary" invocation pattern.

Canary selection:

ID	Recipe	SHA	Rationale
good-simple	custom-html-tiny	435df8fc (main)	Fast, few deps, quick signal
good-significant	lasuite-docs	290a8ad7 (main)	Multi-service, exercises real breadth
bad-false-green	custom-html	71e7326a (v5-stale-docroot)	Already produced RED build #75; pinned fixture

SHAs confirmed from Gitea API on 2026-06-02.

Semantic checks ("teeth") design:

The regression tests assert BOTH exit code AND named tests in results.json stages. This guards against two failure modes:

1. Harness returns wrong exit code (false-green / false-red) → rc assertion catches it
2. A specific assertion is silently removed/vacuated → named test disappears from stages → semantic check catches it

For custom-html-tiny: test_serving (generic install) must appear passing For lasuite-docs: test_serving_and_frontend (install overlay) must appear passing For bad canary: test_content_type (custom functional) must appear failing

File layout:

• tests/regression/conftest.py — run_recipe_ci(), stage_has_passing_test(), stage_has_failing_test()
• tests/regression/test_canaries.py — parametrized @pytest.mark.canary test
• tests/regression/README.md — cadence policy + how to run + how to add

Next step: commit + push, then run good-simple and bad-false-green canaries to get real output. lasuite-docs is slow (10-20 min) so will run it last.

---

Step 1 — initial canary runs (2026-06-02 ~01:28-01:40Z)

bad-false-green run (regression-bad-canary-1)

Command: RECIPE=custom-html REF=71e7326a... SRC=recipe-maintainers/custom-html cc-ci-run runner/run_recipe_ci.py Result: RC=1, custom=FAIL Key output:

• test_content_type_html_and_txt FAILED: ccci-89273b0b.txt Content-Type='application/octet-stream', expected text/plain
• All other tiers (install/upgrade/backup/restore): PASS
• flags: {clean_teardown: True, no_secret_leak: True}
• Confirms: regression test assert rc != 0 will PASS ✓
• Confirms: stage_has_failing_test(results, "custom", "test_content_type") will return True ✓

good-simple run (regression-good-simple-1)

Command: RECIPE=custom-html-tiny REF=435df8fc... SRC=recipe-maintainers/custom-html-tiny cc-ci-run runner/run_recipe_ci.py Result: RC=0, install=pass, upgrade=pass, backup/restore/custom=skip Key output:

• test_serving in install stage: PASSED ✓
• flags: {clean_teardown: True, no_secret_leak: True} ✓
• Confirms: all regression assertions for good-simple will PASS ✓

good-significant run (regression-good-significant-1) [IN PROGRESS]

Started ~01:35Z. Multi-service stack (lasuite-docs + keycloak dep). Image pull in progress. Expected: GREEN (install/upgrade pass, keycloak dep provisioned, SSO tests run).