feat(regression): E2E canary regression suite (tests/regression/) #5

2026-06-02T03:33:47Z

autonomic-bot commented

2026-06-02 03:33:47 +00:00

Summary

Adds a standing, codified regression test suite (tests/regression/) so we can modify the cc-ci server without silently breaking it. Not a prompt — a deterministic pytest artifact runnable on-demand.

What is in this PR

tests/regression/

conftest.py — run_recipe_ci() wrapper + stage assertion helpers (stage_has_passing_test, stage_has_failing_test)
test_canaries.py — 7 parametrized canaries with @pytest.mark.canary / @pytest.mark.canary_fast
README.md — cadence policy, how to run, how to add a canary

tests/custom-html-bkp-bad/ — cc-ci recipe for the backup-tier RED canary (custom-html without backupbot labels → backup yields no artifact → backup=RED)

tests/custom-html-rst-bad/ — cc-ci recipe for the restore-tier RED canary (no pre_backup hook → snapshot has no marker → restore=RED)

Canaries (7 total)

ID	Recipe	Expected	Verified
good-simple	custom-html-tiny @ 435df8fc	GREEN	✓ rc=0, install/upgrade=pass
good-significant	lasuite-docs @ 290a8ad7	GREEN	✓ rc=0, all tiers pass
bad-false-green	custom-html @ 71e7326a (v5-stale-docroot)	RED	✓ rc=1, custom=fail
bad-install	custom-html-tiny @ 4ae88661 (bad image)	RED (install)	✓ rc=1, install=fail
bad-upgrade	custom-html-tiny @ 4ae88661 (bad image)	RED (upgrade)	✓ rc=1, install=pass, upgrade=fail
bad-backup	custom-html-bkp-bad @ b6fe99de	RED (backup)	✓ rc=1, install=pass, backup=fail
bad-restore	custom-html-rst-bad @ 9a73a184	RED (restore)	✓ rc=1, install/backup=pass, restore=fail

How to run

# On cc-ci server (requires abra, Docker, Swarm):
cc-ci-run -m pytest tests/regression/ -m canary -v        # full suite
cc-ci-run -m pytest tests/regression/ -m canary_fast -v   # fast RED subset (~2-3 min each)

Cadence policy

Do NOT run on every commit. These are slow + resource-heavy (lasuite-docs full lifecycle = 10-20 min). Run deliberately: before server releases, polishing passes, on-demand pre-merge. See tests/regression/README.md.

Do not merge

This PR is for operator review only — loops never merge.

## Summary Adds a standing, codified regression test suite (`tests/regression/`) so we can modify the cc-ci server without silently breaking it. Not a prompt — a deterministic pytest artifact runnable on-demand. ## What is in this PR **`tests/regression/`** - `conftest.py` — `run_recipe_ci()` wrapper + stage assertion helpers (`stage_has_passing_test`, `stage_has_failing_test`) - `test_canaries.py` — 7 parametrized canaries with `@pytest.mark.canary` / `@pytest.mark.canary_fast` - `README.md` — cadence policy, how to run, how to add a canary **`tests/custom-html-bkp-bad/`** — cc-ci recipe for the backup-tier RED canary (custom-html without backupbot labels → backup yields no artifact → backup=RED) **`tests/custom-html-rst-bad/`** — cc-ci recipe for the restore-tier RED canary (no pre_backup hook → snapshot has no marker → restore=RED) ## Canaries (7 total) | ID | Recipe | Expected | Verified | |----|--------|----------|----------| | good-simple | custom-html-tiny @ 435df8fc | GREEN | ✓ rc=0, install/upgrade=pass | | good-significant | lasuite-docs @ 290a8ad7 | GREEN | ✓ rc=0, all tiers pass | | bad-false-green | custom-html @ 71e7326a (v5-stale-docroot) | RED | ✓ rc=1, custom=fail | | bad-install | custom-html-tiny @ 4ae88661 (bad image) | RED (install) | ✓ rc=1, install=fail | | bad-upgrade | custom-html-tiny @ 4ae88661 (bad image) | RED (upgrade) | ✓ rc=1, install=pass, upgrade=fail | | bad-backup | custom-html-bkp-bad @ b6fe99de | RED (backup) | ✓ rc=1, install=pass, backup=fail | | bad-restore | custom-html-rst-bad @ 9a73a184 | RED (restore) | ✓ rc=1, install/backup=pass, restore=fail | ## How to run ```bash # On cc-ci server (requires abra, Docker, Swarm): cc-ci-run -m pytest tests/regression/ -m canary -v # full suite cc-ci-run -m pytest tests/regression/ -m canary_fast -v # fast RED subset (~2-3 min each) ``` ## Cadence policy **Do NOT run on every commit.** These are slow + resource-heavy (lasuite-docs full lifecycle = 10-20 min). Run deliberately: before server releases, polishing passes, on-demand pre-merge. See `tests/regression/README.md`. ## Do not merge This PR is for operator review only — loops never merge.

autonomic-bot added 1 commit 2026-06-02 03:33:47 +00:00

feat(regression): add E2E canary regression suite (tests/regression/)

continuous-integration/drone/push Build is failing

Details

73427d6e36

Seven canaries prove both halves of the server's job:
- GREEN: good apps are reported healthy (good-simple + good-significant)
- RED: broken apps are caught at intended tier (false-green guard + 4 per-tier)

Fixtures: custom-html-bkp-bad (backup tier RED) + custom-html-rst-bad (restore tier RED).
All 7 canaries verified on live server (see STATUS-regression.md for artifacts).

Not wired to per-commit CI — run on-demand: pytest -m canary tests/regression/

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

autonomic-bot referenced this issue from a commit

2026-06-02 03:35:02 +00:00

claim(D-final): all 7 canaries verified + PR#5 opened — FINAL gate claim

autonomic-bot referenced this issue from a commit

2026-06-02 03:38:26 +00:00

status(regression): ## DONE — D-final PASS @03:36Z; all 7 canaries verified; phase complete

autonomic-bot commented

2026-06-11 21:05:24 +00:00

Closing as superseded. The canary suite (tests/regression/) landed directly on main 2026-06-02 (regression phase, Adversary-verified) — already identical in main. Branch is hundreds of commits behind; merging would revert later work (~12.9k deletions). Not merging.

Closing as **superseded**. The canary suite (tests/regression/) landed directly on main 2026-06-02 (regression phase, Adversary-verified) — already identical in main. Branch is hundreds of commits behind; merging would revert later work (~12.9k deletions). Not merging.

autonomic-bot closed this pull request

2026-06-11 21:05:24 +00:00

autonomic-bot referenced this issue from a commit

2026-06-17 06:45:44 +00:00

feat(canon): M1.2 release-tag trigger + faithful mirror-sync in the weekly sweep (§2.C/§2.D)

autonomic-bot referenced this issue from a commit

2026-06-17 07:07:46 +00:00

claim(M1): canonical sweep machinery built + live-proven on custom-html

autonomic-bot referenced this issue from a commit

2026-07-08 23:29:11 +00:00

review(redfix): post-reboot re-confirmation #5 — DONE stands, no VETO. Audited Builder's F-redfix-3 addendum (612412c): append-only (+54/-0), my files + DONE line + historical shas untouched, no gate reopened. Ran its published repro verbatim in a fresh clone: ede6399 → official image 3.5.3 + 0 sidekiq, both EXPECTED met. Ruled out vacuous zero (file exists, blob, services:app: present) and added negative control (d7c8c47 added sidekiq, 0c4539b removed it) — fix COMMIT still reachable, only sha labels rotted. F-redfix-3 remedy verified, not just asserted

autonomic-bot referenced this issue from a commit