feat(regression): E2E canary regression suite — operator review #4

New Issue

2026-06-02T02:19:04Z

autonomic-bot commented

2026-06-02 02:19:04 +00:00

Summary

This represents the server regression canaries phase — a standing pytest suite that gives the cc-ci server codified E2E self-tests.

What was built

tests/regression/ — 7 parametrized canaries (@pytest.mark.canary + @pytest.mark.canary_fast):

Canary	Recipe	Expected	Status
good-simple	custom-html-tiny @ 435df8fc	GREEN	✓ Verified
good-significant	lasuite-docs @ 290a8ad7	GREEN	Re-running (upgrade flakiness)
bad-false-green	custom-html @ 71e7326a (v5-stale-docroot)	RED (custom=fail)	✓ Verified
bad-install	custom-html-tiny @ 4ae88661	RED (install=fail)	✓ Verified
bad-upgrade	custom-html-tiny @ 4ae88661	RED (upgrade=fail)	✓ Verified
bad-backup	custom-html-bkp-bad @ b6fe99de	RED (backup=fail)	✓ Verified
bad-restore	custom-html-rst-bad @ 9a73a184	RED (restore=fail)	✓ Verified

How to run

# Fast RED canaries (~2-3 min each):
cd /root/builder-clone && cc-ci-run -m pytest tests/regression/ -m canary_fast -v

# Full suite (includes lasuite-docs, 10-20 min):
cc-ci-run -m pytest tests/regression/ -m canary -v

Related recipe fixtures on Gitea

recipe-maintainers/custom-html-tiny branch regression-bad-image — bad image for install/upgrade RED
recipe-maintainers/custom-html branch v5-stale-docroot — existing bad MIME type fixture
recipe-maintainers/custom-html-bkp-bad — no pre_backup hook, test_backup.py fails
recipe-maintainers/custom-html-rst-bad — no pre_backup hook, ops.py+test_restore.py

Definition of Done (plan § DoD)

✓ tests/regression/ suite committed (7 tests, 0 collection errors)
✓ good-simple GREEN — install=pass, upgrade=pass, test_serving PASS in install stage
✓ bad-false-green RED — custom=fail, test_content_type fails (content-type guard has teeth)
✓ 4 per-tier RED canaries: each asserts RED at the intended tier with prior tiers PASS
✓ README.md with cadence policy and how to add a canary
✓ NOT merged — this PR is for operator review only (loops never merge)

good-significant (lasuite-docs) upgrade had a transient WOPI convergence race on first run; re-running now to confirm.

🤖 Generated with Claude Code

## Summary This represents the **server regression canaries** phase — a standing pytest suite that gives the cc-ci server codified E2E self-tests. ### What was built `tests/regression/` — 7 parametrized canaries (`@pytest.mark.canary` + `@pytest.mark.canary_fast`): | Canary | Recipe | Expected | Status | |--------|--------|----------|--------| | good-simple | custom-html-tiny @ 435df8fc | GREEN | ✓ Verified | | good-significant | lasuite-docs @ 290a8ad7 | GREEN | Re-running (upgrade flakiness) | | bad-false-green | custom-html @ 71e7326a (v5-stale-docroot) | RED (custom=fail) | ✓ Verified | | bad-install | custom-html-tiny @ 4ae88661 | RED (install=fail) | ✓ Verified | | bad-upgrade | custom-html-tiny @ 4ae88661 | RED (upgrade=fail) | ✓ Verified | | bad-backup | custom-html-bkp-bad @ b6fe99de | RED (backup=fail) | ✓ Verified | | bad-restore | custom-html-rst-bad @ 9a73a184 | RED (restore=fail) | ✓ Verified | ### How to run ```bash # Fast RED canaries (~2-3 min each): cd /root/builder-clone && cc-ci-run -m pytest tests/regression/ -m canary_fast -v # Full suite (includes lasuite-docs, 10-20 min): cc-ci-run -m pytest tests/regression/ -m canary -v ``` ### Related recipe fixtures on Gitea - `recipe-maintainers/custom-html-tiny` branch `regression-bad-image` — bad image for install/upgrade RED - `recipe-maintainers/custom-html` branch `v5-stale-docroot` — existing bad MIME type fixture - `recipe-maintainers/custom-html-bkp-bad` — no pre_backup hook, test_backup.py fails - `recipe-maintainers/custom-html-rst-bad` — no pre_backup hook, ops.py+test_restore.py ### Definition of Done (plan § DoD) 1. ✓ `tests/regression/` suite committed (7 tests, 0 collection errors) 2. ✓ good-simple GREEN — install=pass, upgrade=pass, test_serving PASS in install stage 3. ✓ bad-false-green RED — custom=fail, test_content_type fails (content-type guard has teeth) 4. ✓ 4 per-tier RED canaries: each asserts RED at the intended tier with prior tiers PASS 5. ✓ README.md with cadence policy and how to add a canary 6. ✓ NOT merged — this PR is for operator review only (loops never merge) good-significant (lasuite-docs) upgrade had a transient WOPI convergence race on first run; re-running now to confirm. --- 🤖 Generated with [Claude Code](https://claude.com/claude-code)

notplants closed this issue

2026-06-11 21:10:58 +00:00

autonomic-bot referenced this issue from a commit

2026-06-17 01:48:42 +00:00

review(prevb): M2 cold-verify IN FLIGHT — discourse #4 !testme GREEN confirmed via gitea API (Drone 717, real live-image teeth, lint=non-gating rung); 3 spot-checks dynamic-base confirmed; my own cryptpad re-run in flight

autonomic-bot referenced this issue from a commit

2026-06-17 01:50:03 +00:00

review(prevb): M2 PASS — discourse #4 !testme GREEN in real CI (Drone 717, live-image teeth=official 3.5.3, lint non-gating); 3 spot-checks + own cryptpad re-run confirm dynamic base; public surface secret-clean; nothing merged. Both M1+M2 PASS, no VETO → Builder may DONE

autonomic-bot referenced this issue from a commit

2026-06-17 04:58:50 +00:00

claim(M2): samever proven in real CI — step-back base<head, version-bump unaffected, discourse #4 + hedgedoc spot-check

autonomic-bot referenced this issue from a commit