plan: regression canaries are milestone-cadence (polish/review/release), not per-commit
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
@ -61,10 +61,17 @@ For each good canary the test drives the real cold full suite and asserts the *o
|
||||
- **Determinism:** pin canary recipe versions/SHAs and the bad-fixture SHA. Record run artifacts
|
||||
(build URL, logs) on failure for triage.
|
||||
|
||||
## How it runs (dogfood)
|
||||
## How it runs — CADENCE POLICY (important)
|
||||
**Do NOT run these canaries on every commit/PR.** They are slow + resource-heavy (full lifecycle on
|
||||
lasuite-docs is minutes, needs the live server/abra/Swarm). Run them **deliberately at milestones**:
|
||||
- **polishing passes**, **code reviews**, and **releases** of the cc-ci server — i.e. before we trust a
|
||||
batch of server changes, not on each incremental commit.
|
||||
- On-demand: `pytest -m canary` against the live cc-ci server (from the orchestrator or the host).
|
||||
- Dogfood: cc-ci is itself enrolled in `!testme`, so a server-change PR on the cc-ci repo can trigger
|
||||
the canary run automatically — a server regression shows up as a RED check on the very PR that caused it.
|
||||
- They are explicitly **opt-in** (the `@pytest.mark.canary` marker keeps them out of any fast/default
|
||||
run). If wired to `!testme` on the cc-ci repo, gate it behind a deliberate trigger (e.g. a
|
||||
`run-canaries` label or a `!testme --canary`), **not** an automatic run on every cc-ci PR.
|
||||
- Document this cadence in `tests/regression/README.md` so future contributors don't wire it into the
|
||||
per-commit path.
|
||||
|
||||
## Definition of Done (Adversary-verified)
|
||||
1. `tests/regression/` pytest suite exists and is committed (cc-ci repo PR).
|
||||
|
||||
Reference in New Issue
Block a user