a2a6eea757
Some checks failed
continuous-integration/drone/push Build is failing
- tests/regression/test_canaries.py: replace `from .conftest import ...` (relative import fails when not a package) with sys.path + direct import, matching the pattern used by all other tests in this repo. - Delete machine-docs/BUILDER-INBOX.md (Adversary inbox consumed). - Update STATUS-regression.md + JOURNAL-regression.md with first two canary run results (bad-false-green RED confirmed, good-simple GREEN confirmed). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
77 lines
3.6 KiB
Markdown
77 lines
3.6 KiB
Markdown
# JOURNAL — server regression canaries phase (Builder)
|
|
|
|
**Phase:** server regression canaries
|
|
**Started:** 2026-06-02
|
|
|
|
---
|
|
|
|
## Step 0 — phase kickoff and design (2026-06-02)
|
|
|
|
**Context:** Mirror phase (plan-mirror-enroll-all-recipes.md) completed DONE at 2026-06-02T01:16Z.
|
|
Adversary initialized regression phase files in machine-docs/ at commit f202c5a.
|
|
|
|
**Decision: run regression tests ON cc-ci, not from the orchestrator**
|
|
|
|
The regression tests call `run_recipe_ci.py` which uses abra/docker/swarm — these only exist on
|
|
cc-ci. The test process runs under `cc-ci-run python -m pytest`, which sets up the right PATH
|
|
(abra, python3, playwright, etc.). The test then invokes `run_recipe_ci.py` as a subprocess using
|
|
`sys.executable` (inherits the same python3 from cc-ci-run).
|
|
|
|
The README.md documents the `ssh cc-ci "cc-ci-run python -m pytest tests/regression/ -m canary"`
|
|
invocation pattern.
|
|
|
|
**Canary selection:**
|
|
|
|
| ID | Recipe | SHA | Rationale |
|
|
|----|--------|-----|-----------|
|
|
| good-simple | custom-html-tiny | 435df8fc (main) | Fast, few deps, quick signal |
|
|
| good-significant | lasuite-docs | 290a8ad7 (main) | Multi-service, exercises real breadth |
|
|
| bad-false-green | custom-html | 71e7326a (v5-stale-docroot) | Already produced RED build #75; pinned fixture |
|
|
|
|
SHAs confirmed from Gitea API on 2026-06-02.
|
|
|
|
**Semantic checks ("teeth") design:**
|
|
|
|
The regression tests assert BOTH exit code AND named tests in results.json stages. This guards
|
|
against two failure modes:
|
|
1. Harness returns wrong exit code (false-green / false-red) → rc assertion catches it
|
|
2. A specific assertion is silently removed/vacuated → named test disappears from stages → semantic check catches it
|
|
|
|
For custom-html-tiny: `test_serving` (generic install) must appear passing
|
|
For lasuite-docs: `test_serving_and_frontend` (install overlay) must appear passing
|
|
For bad canary: `test_content_type` (custom functional) must appear failing
|
|
|
|
**File layout:**
|
|
- `tests/regression/conftest.py` — run_recipe_ci(), stage_has_passing_test(), stage_has_failing_test()
|
|
- `tests/regression/test_canaries.py` — parametrized @pytest.mark.canary test
|
|
- `tests/regression/README.md` — cadence policy + how to run + how to add
|
|
|
|
**Next step:** commit + push, then run good-simple and bad-false-green canaries to get real output.
|
|
lasuite-docs is slow (10-20 min) so will run it last.
|
|
|
|
---
|
|
|
|
## Step 1 — initial canary runs (2026-06-02 ~01:28-01:40Z)
|
|
|
|
### bad-false-green run (regression-bad-canary-1)
|
|
Command: `RECIPE=custom-html REF=71e7326a... SRC=recipe-maintainers/custom-html cc-ci-run runner/run_recipe_ci.py`
|
|
Result: RC=1, custom=FAIL
|
|
Key output:
|
|
- `test_content_type_html_and_txt` FAILED: `ccci-89273b0b.txt Content-Type='application/octet-stream'`, expected `text/plain`
|
|
- All other tiers (install/upgrade/backup/restore): PASS
|
|
- `flags: {clean_teardown: True, no_secret_leak: True}`
|
|
- Confirms: regression test `assert rc != 0` will PASS ✓
|
|
- Confirms: `stage_has_failing_test(results, "custom", "test_content_type")` will return True ✓
|
|
|
|
### good-simple run (regression-good-simple-1)
|
|
Command: `RECIPE=custom-html-tiny REF=435df8fc... SRC=recipe-maintainers/custom-html-tiny cc-ci-run runner/run_recipe_ci.py`
|
|
Result: RC=0, install=pass, upgrade=pass, backup/restore/custom=skip
|
|
Key output:
|
|
- `test_serving` in install stage: PASSED ✓
|
|
- `flags: {clean_teardown: True, no_secret_leak: True}` ✓
|
|
- Confirms: all regression assertions for good-simple will PASS ✓
|
|
|
|
### good-significant run (regression-good-significant-1) [IN PROGRESS]
|
|
Started ~01:35Z. Multi-service stack (lasuite-docs + keycloak dep). Image pull in progress.
|
|
Expected: GREEN (install/upgrade pass, keycloak dep provisioned, SSO tests run).
|