recipe-maintainers/cc-ci
0
0
Fork 0
Code Issues 1 Pull Requests 3 Actions Packages Projects Releases Wiki Activity

feat(regression): E2E canary regression suite (tests/regression/) #5

Open
autonomic-bot wants to merge 1 commits from regression-canaries into main
pull from: regression-canaries
merge into: recipe-maintainers:main
Conversation 0 Commits 1 Files Changed 10 +704
autonomic-bot commented 2026-06-02 03:33:47 +00:00
Owner
Copy Link

Summary

Adds a standing, codified regression test suite (tests/regression/) so we can modify the cc-ci server without silently breaking it. Not a prompt — a deterministic pytest artifact runnable on-demand.

What is in this PR

tests/regression/

  • conftest.py — run_recipe_ci() wrapper + stage assertion helpers (stage_has_passing_test, stage_has_failing_test)
  • test_canaries.py — 7 parametrized canaries with @pytest.mark.canary / @pytest.mark.canary_fast
  • README.md — cadence policy, how to run, how to add a canary

tests/custom-html-bkp-bad/ — cc-ci recipe for the backup-tier RED canary (custom-html without backupbot labels → backup yields no artifact → backup=RED)

tests/custom-html-rst-bad/ — cc-ci recipe for the restore-tier RED canary (no pre_backup hook → snapshot has no marker → restore=RED)

Canaries (7 total)

ID Recipe Expected Verified
good-simple custom-html-tiny @ 435df8fc GREEN ✓ rc=0, install/upgrade=pass
good-significant lasuite-docs @ 290a8ad7 GREEN ✓ rc=0, all tiers pass
bad-false-green custom-html @ 71e7326a (v5-stale-docroot) RED ✓ rc=1, custom=fail
bad-install custom-html-tiny @ 4ae88661 (bad image) RED (install) ✓ rc=1, install=fail
bad-upgrade custom-html-tiny @ 4ae88661 (bad image) RED (upgrade) ✓ rc=1, install=pass, upgrade=fail
bad-backup custom-html-bkp-bad @ b6fe99de RED (backup) ✓ rc=1, install=pass, backup=fail
bad-restore custom-html-rst-bad @ 9a73a184 RED (restore) ✓ rc=1, install/backup=pass, restore=fail

How to run

# On cc-ci server (requires abra, Docker, Swarm):
cc-ci-run -m pytest tests/regression/ -m canary -v        # full suite
cc-ci-run -m pytest tests/regression/ -m canary_fast -v   # fast RED subset (~2-3 min each)

Cadence policy

Do NOT run on every commit. These are slow + resource-heavy (lasuite-docs full lifecycle = 10-20 min). Run deliberately: before server releases, polishing passes, on-demand pre-merge. See tests/regression/README.md.

Do not merge

This PR is for operator review only — loops never merge.

## Summary Adds a standing, codified regression test suite (`tests/regression/`) so we can modify the cc-ci server without silently breaking it. Not a prompt — a deterministic pytest artifact runnable on-demand. ## What is in this PR **`tests/regression/`** - `conftest.py` — `run_recipe_ci()` wrapper + stage assertion helpers (`stage_has_passing_test`, `stage_has_failing_test`) - `test_canaries.py` — 7 parametrized canaries with `@pytest.mark.canary` / `@pytest.mark.canary_fast` - `README.md` — cadence policy, how to run, how to add a canary **`tests/custom-html-bkp-bad/`** — cc-ci recipe for the backup-tier RED canary (custom-html without backupbot labels → backup yields no artifact → backup=RED) **`tests/custom-html-rst-bad/`** — cc-ci recipe for the restore-tier RED canary (no pre_backup hook → snapshot has no marker → restore=RED) ## Canaries (7 total) | ID | Recipe | Expected | Verified | |----|--------|----------|----------| | good-simple | custom-html-tiny @ 435df8fc | GREEN | ✓ rc=0, install/upgrade=pass | | good-significant | lasuite-docs @ 290a8ad7 | GREEN | ✓ rc=0, all tiers pass | | bad-false-green | custom-html @ 71e7326a (v5-stale-docroot) | RED | ✓ rc=1, custom=fail | | bad-install | custom-html-tiny @ 4ae88661 (bad image) | RED (install) | ✓ rc=1, install=fail | | bad-upgrade | custom-html-tiny @ 4ae88661 (bad image) | RED (upgrade) | ✓ rc=1, install=pass, upgrade=fail | | bad-backup | custom-html-bkp-bad @ b6fe99de | RED (backup) | ✓ rc=1, install=pass, backup=fail | | bad-restore | custom-html-rst-bad @ 9a73a184 | RED (restore) | ✓ rc=1, install/backup=pass, restore=fail | ## How to run ```bash # On cc-ci server (requires abra, Docker, Swarm): cc-ci-run -m pytest tests/regression/ -m canary -v # full suite cc-ci-run -m pytest tests/regression/ -m canary_fast -v # fast RED subset (~2-3 min each) ``` ## Cadence policy **Do NOT run on every commit.** These are slow + resource-heavy (lasuite-docs full lifecycle = 10-20 min). Run deliberately: before server releases, polishing passes, on-demand pre-merge. See `tests/regression/README.md`. ## Do not merge This PR is for operator review only — loops never merge.
autonomic-bot added 1 commit 2026-06-02 03:33:47 +00:00
feat(regression): add E2E canary regression suite (tests/regression/)
Some checks failed
continuous-integration/drone/push Build is failing
Details
73427d6e36 
Seven canaries prove both halves of the server's job:
- GREEN: good apps are reported healthy (good-simple + good-significant)
- RED: broken apps are caught at intended tier (false-green guard + 4 per-tier)

Fixtures: custom-html-bkp-bad (backup tier RED) + custom-html-rst-bad (restore tier RED).
All 7 canaries verified on live server (see STATUS-regression.md for artifacts).

Not wired to per-commit CI — run on-demand: pytest -m canary tests/regression/

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Some checks failed
continuous-integration/drone/push Build is failing
Details
Checking for merge conflicts ...
View command line instructions

Checkout

From your project repository, check out a new branch and test the changes.
git fetch -u origin regression-canaries:regression-canaries
git checkout regression-canaries
Sign in to join this conversation.
Reviewers
No Reviewers
Labels
No items
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
autonomic autonomic-bot notplants trav
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: recipe-maintainers/cc-ci#5
No description provided.
Delete Branch "regression-canaries"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?