feat(regression): E2E canary regression suite (tests/regression/) #5

Closed

autonomic-bot wants to merge 1 commits from regression-canaries into main

pull from: regression-canaries

merge into: :main

:main

:config/deepseek-flash-zen

:config/deepseek-flash-subagents

:policy/pr-then-merge

:fix/plausible-format

:fix/plausible-v3-custom-tests

:test/upgrade-base-prefer-newest-release-20260810

:test/discourse-upgrade-base-floor-20260804

:test/wordpress-enroll-20260803

:test/lasuite-meet-stale-test-20260803

:test/lasuite-docs-stale-test-20260803

:chore/server-flake-update-20260803

:redfix-m2-harness

:feat/libredesk-tests

:phase-lvl5

:fix/converged-oneshot

:restructure/recipe-custom

:restructure/concurrency

:fix/converge-paused-update

:test/plausible-psql-quiet

:fix/converge-rolling-update

:fix/lint-and-concurrent-runs

:test/plausible-upgrade-base-3.0.1

:chore/runner-capacity-2

:feat/reports-pr-status-proxy

:feat/expected-na-and-tiny-functional

:regression-canaries

:v6-custom-html-mime

:v6-custom-html-docroot

:v6-matrix-synapse-real-upgrade-state

:terraform-hetzner

:scratch/m3-demo

Conversation 1 Commits 1 Files Changed 10

+704

autonomic-bot commented 2026-06-02 03:33:47 +00:00

Owner

Copy Link

Copy Source

Summary

Adds a standing, codified regression test suite (tests/regression/) so we can modify the cc-ci server without silently breaking it. Not a prompt — a deterministic pytest artifact runnable on-demand.

What is in this PR

tests/regression/

• conftest.py — run_recipe_ci() wrapper + stage assertion helpers (stage_has_passing_test, stage_has_failing_test)
• test_canaries.py — 7 parametrized canaries with @pytest.mark.canary / @pytest.mark.canary_fast
• README.md — cadence policy, how to run, how to add a canary

tests/custom-html-bkp-bad/ — cc-ci recipe for the backup-tier RED canary (custom-html without backupbot labels → backup yields no artifact → backup=RED)

tests/custom-html-rst-bad/ — cc-ci recipe for the restore-tier RED canary (no pre_backup hook → snapshot has no marker → restore=RED)

Canaries (7 total)

ID	Recipe	Expected	Verified
good-simple	custom-html-tiny @ 435df8fc	GREEN	✓ rc=0, install/upgrade=pass
good-significant	lasuite-docs @ 290a8ad7	GREEN	✓ rc=0, all tiers pass
bad-false-green	custom-html @ 71e7326a (v5-stale-docroot)	RED	✓ rc=1, custom=fail
bad-install	custom-html-tiny @ 4ae88661 (bad image)	RED (install)	✓ rc=1, install=fail
bad-upgrade	custom-html-tiny @ 4ae88661 (bad image)	RED (upgrade)	✓ rc=1, install=pass, upgrade=fail
bad-backup	custom-html-bkp-bad @ b6fe99de	RED (backup)	✓ rc=1, install=pass, backup=fail
bad-restore	custom-html-rst-bad @ 9a73a184	RED (restore)	✓ rc=1, install/backup=pass, restore=fail

How to run

# On cc-ci server (requires abra, Docker, Swarm):
cc-ci-run -m pytest tests/regression/ -m canary -v        # full suite
cc-ci-run -m pytest tests/regression/ -m canary_fast -v   # fast RED subset (~2-3 min each)

Cadence policy

Do NOT run on every commit. These are slow + resource-heavy (lasuite-docs full lifecycle = 10-20 min). Run deliberately: before server releases, polishing passes, on-demand pre-merge. See tests/regression/README.md.

Do not merge

This PR is for operator review only — loops never merge.

## Summary Adds a standing, codified regression test suite (`tests/regression/`) so we can modify the cc-ci server without silently breaking it. Not a prompt — a deterministic pytest artifact runnable on-demand. ## What is in this PR **`tests/regression/`** - `conftest.py` — `run_recipe_ci()` wrapper + stage assertion helpers (`stage_has_passing_test`, `stage_has_failing_test`) - `test_canaries.py` — 7 parametrized canaries with `@pytest.mark.canary` / `@pytest.mark.canary_fast` - `README.md` — cadence policy, how to run, how to add a canary **`tests/custom-html-bkp-bad/`** — cc-ci recipe for the backup-tier RED canary (custom-html without backupbot labels → backup yields no artifact → backup=RED) **`tests/custom-html-rst-bad/`** — cc-ci recipe for the restore-tier RED canary (no pre_backup hook → snapshot has no marker → restore=RED) ## Canaries (7 total) | ID | Recipe | Expected | Verified | |----|--------|----------|----------| | good-simple | custom-html-tiny @ 435df8fc | GREEN | ✓ rc=0, install/upgrade=pass | | good-significant | lasuite-docs @ 290a8ad7 | GREEN | ✓ rc=0, all tiers pass | | bad-false-green | custom-html @ 71e7326a (v5-stale-docroot) | RED | ✓ rc=1, custom=fail | | bad-install | custom-html-tiny @ 4ae88661 (bad image) | RED (install) | ✓ rc=1, install=fail | | bad-upgrade | custom-html-tiny @ 4ae88661 (bad image) | RED (upgrade) | ✓ rc=1, install=pass, upgrade=fail | | bad-backup | custom-html-bkp-bad @ b6fe99de | RED (backup) | ✓ rc=1, install=pass, backup=fail | | bad-restore | custom-html-rst-bad @ 9a73a184 | RED (restore) | ✓ rc=1, install/backup=pass, restore=fail | ## How to run ```bash # On cc-ci server (requires abra, Docker, Swarm): cc-ci-run -m pytest tests/regression/ -m canary -v # full suite cc-ci-run -m pytest tests/regression/ -m canary_fast -v # fast RED subset (~2-3 min each) ``` ## Cadence policy **Do NOT run on every commit.** These are slow + resource-heavy (lasuite-docs full lifecycle = 10-20 min). Run deliberately: before server releases, polishing passes, on-demand pre-merge. See `tests/regression/README.md`. ## Do not merge This PR is for operator review only — loops never merge.

autonomic-bot added 1 commit 2026-06-02 03:33:47 +00:00

feat(regression): add E2E canary regression suite (tests/regression/) ... 73427d6e36

Seven canaries prove both halves of the server's job:
- GREEN: good apps are reported healthy (good-simple + good-significant)
- RED: broken apps are caught at intended tier (false-green guard + 4 per-tier)

Fixtures: custom-html-bkp-bad (backup tier RED) + custom-html-rst-bad (restore tier RED).
All 7 canaries verified on live server (see STATUS-regression.md for artifacts).

Not wired to per-commit CI — run on-demand: pytest -m canary tests/regression/

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

autonomic-bot referenced this issue from a commit 2026-06-02 03:35:02 +00:00

claim(D-final): all 7 canaries verified + PR#5 opened — FINAL gate claim

autonomic-bot referenced this issue from a commit 2026-06-02 03:38:26 +00:00

status(regression): ## DONE — D-final PASS @03:36Z; all 7 canaries verified; phase complete

autonomic-bot commented 2026-06-11 21:05:24 +00:00

Author

Owner

Copy Link

Copy Source

Closing as superseded. The canary suite (tests/regression/) landed directly on main 2026-06-02 (regression phase, Adversary-verified) — already identical in main. Branch is hundreds of commits behind; merging would revert later work (~12.9k deletions). Not merging.

Closing as **superseded**. The canary suite (tests/regression/) landed directly on main 2026-06-02 (regression phase, Adversary-verified) — already identical in main. Branch is hundreds of commits behind; merging would revert later work (~12.9k deletions). Not merging.

autonomic-bot closed this pull request 2026-06-11 21:05:24 +00:00

autonomic-bot referenced this issue from a commit 2026-06-17 06:45:44 +00:00

feat(canon): M1.2 release-tag trigger + faithful mirror-sync in the weekly sweep (§2.C/§2.D)

autonomic-bot referenced this issue from a commit 2026-06-17 07:07:46 +00:00

claim(M1): canonical sweep machinery built + live-proven on custom-html

autonomic-bot referenced this issue from a commit 2026-07-08 23:29:11 +00:00

review(redfix): post-reboot re-confirmation #5 — DONE stands, no VETO. Audited Builder's F-redfix-3 addendum (612412c): append-only (+54/-0), my files + DONE line + historical shas untouched, no gate reopened. Ran its published repro verbatim in a fresh clone: ede6399 → official image 3.5.3 + 0 sidekiq, both EXPECTED met. Ruled out vacuous zero (file exists, blob, services:app: present) and added negative control (d7c8c47 added sidekiq, 0c4539b removed it) — fix COMMIT still reachable, only sha labels rotted. F-redfix-3 remedy verified, not just asserted

autonomic-bot referenced this issue from a commit 2026-07-08 23:30:06 +00:00

journal(redfix): wake #5 — consumed Adversary re-confirmation #5 (audit of my F-redfix-3 addendum: clean, remedy verified, fix commit still reachable). DONE stands, no VETO, no action. Loop stopped

autonomic-bot referenced this issue from a commit 2026-07-08 23:36:01 +00:00

review(redfix): post-reboot re-confirmation #6 — DONE stands, no VETO. Zero delta since #5 (only a Builder JOURNAL append). Break-it probe on evidence-anchor rot: cold bare-cloned all 3 remaining recipe mirrors — mattermost-lts 4ca7f418, gitea a0f2db8, bluesky-pds 4987ba9 all REACHABLE with subjects matching their asserted fixes, and harness branch redfix-m2-harness still pins exactly 07fc6d4 (mumble) with 61211db (keycloak) reachable. Rot confined to discourse (F-redfix-3, CLOSED). All findings closed

Pull request closed

Please reopen this pull request to perform a merge.

Sign in to join this conversation.

Reviewers

No Reviewers

Labels

No items

No labels

Milestone

No items

No Milestone

Projects

Clear projects

No projects

Assignees

Clear assignees

autonomic autonomic-bot kawaiipunk (KawaiiPunk) notplants trav (Trav Fryer)

No Assignees

1 Participants

Notifications

Subscribe

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: recipe-maintainers/cc-ci#5