skills: point every test-editing path at tests/STYLE.md
/recipe-upgrade --with-tests, /ci-test-review and /cc-ci-tests-update all author test changes, and all three had only 'never weaken a test' as guidance. That did not stop the plausible failure: the fixture INSERTed rows into the app's database, which was correct for v2 and silently wrong for v3, where a site must belong to a team. Events were acked 202 and discarded; the recipe sat RED for six weeks. The rule that would have prevented it — set state up through the app's own interface, not its database — now lives in tests/STYLE.md in the cc-ci repo, and each of the three paths is told to read it before editing a test.
This commit is contained in:
@@ -79,6 +79,10 @@ For each real (non-flaky) finding, write the actual fix and open a PR. **Never m
|
||||
it handles the mirror to `git.autonomic.zone/recipe-maintainers/<recipe>` (upstream
|
||||
`git.coopcloud.tech`). Keep the change **bounded** to the diagnosed root cause; don't rewrite the
|
||||
recipe.
|
||||
- **Before editing any test, read `tests/STYLE.md` in the cc-ci repo.** It encodes the rules a test
|
||||
change must satisfy — set state up through the app's interface rather than its database, gate on
|
||||
version instead of branching, correct the fixture/wait but NEVER the assertion, and diagnose from
|
||||
the app's own telemetry before concluding a test is stale.
|
||||
- **CI-server-side fix → cc-ci PR.** Branch the cc-ci product repo
|
||||
(`recipe-maintainers/cc-ci`), apply the fix, and open the PR via the Gitea API (use the
|
||||
`GITEA_*` creds from `/srv/cc-ci/.testenv`). **Single-writer discipline:** work on a dedicated
|
||||
|
||||
@@ -312,6 +312,12 @@ test change, and a test change is **gated by `--with-tests`**:
|
||||
Do **NOT** modify any test. Report `SUCCESS-PENDING-TESTS` (recipe PR open; `!testme` red on a
|
||||
stale test; operator to decide).
|
||||
- **`--with-tests` — open + verify a cc-ci test PR.** Make it the `ci-test-review` way:
|
||||
0. **READ `tests/STYLE.md` in the cc-ci repo FIRST.** It is the rulebook for changing a test, and
|
||||
it is written against the failures this pipeline has actually produced. The two that matter most
|
||||
here: **set state up through the app's own interface, never its database** (a plausible fixture
|
||||
that INSERTed rows passed on v2 and silently broke on v3, holding the recipe RED for six weeks),
|
||||
and **gate on version rather than writing a fixture that supports both** — old-version tests can
|
||||
simply be deleted, since the older version is only exercised through the upgrade tier.
|
||||
1. Branch `recipe-maintainers/cc-ci` in a **separate clone** (single-writer: never push `main`,
|
||||
never touch the build loops' `/cc-ci` `/cc-ci-adv` clones); update the test/overlay.
|
||||
2. **Verify the recipe upgrade WITH the updated test applied.** `!testme` on the recipe PR uses the
|
||||
|
||||
@@ -85,6 +85,14 @@ failure (AI — this is the `ci-test-review` step-3 diagnosis):
|
||||
|
||||
### 2. For each stale test — author the minimal test update (AI; never weaken)
|
||||
|
||||
> **Read `tests/STYLE.md` in the cc-ci repo before writing the update.** It is the rulebook for test
|
||||
> changes, written from failures this pipeline actually produced. Most load-bearing: set state up
|
||||
> through the app's **own interface, never its database** (a plausible fixture that INSERTed rows
|
||||
> passed on v2 and silently broke on v3 — 202 acks, rows in postgres, nothing ingested — and held the
|
||||
> recipe RED for six weeks), **gate on version rather than supporting both** (old-version tests can be
|
||||
> deleted; the older version is only exercised via the upgrade tier), and correct the fixture or the
|
||||
> wait but **never the assertion**.
|
||||
|
||||
Work on **one recipe at a time** (serialize — each verification deploys a recipe on the shared
|
||||
Swarm). For each `STALE_TESTS` entry:
|
||||
|
||||
|
||||
Reference in New Issue
Block a user