cc-ci/machine-docs/REVIEW-regression.md

# REVIEW — server regression canaries phase (Adversary ledger)

**Phase:** server regression canaries (codified E2E self-tests)
**SSOT:** `/srv/cc-ci/cc-ci-plan/plan-server-regression-canaries.md`
**Adversary loop started:** 2026-06-02T01:15Z
**Repo:** git.autonomic.zone/recipe-maintainers/cc-ci
**Adversary clone:** /srv/cc-ci/cc-ci-adv

---

## D-gate verdicts

### D-final: PASS @2026-06-02T03:36Z — all 7 canaries cold-verified; PR#5 open; all DoD items met

**Cold verification result: PASS**

All DoD items independently verified (cold shell, Adversary clone, no cached state):

**DoD#1 — tests/regression/ committed:**
- `cc-ci-run -m pytest tests/regression/ --collect-only -q` on cc-ci from PR branch: 7 tests collected ✓
- Files present on `regression-canaries` branch: `conftest.py`, `test_canaries.py`, `README.md`, plus `tests/custom-html-bkp-bad/` and `tests/custom-html-rst-bad/` ✓

**DoD#2 — both good canaries GREEN with semantic assertion teeth:**
- `good-simple` (regression-good-simple-1, SHA `435df8fc`): `install=pass, upgrade=pass`, `test_serving` PASS in install stage ✓
  - Teeth: if `test_serving` removed → `stage_has_passing_test("install","test_serving")` → False → assert fires ✓
- `good-significant` (regression-good-significant-2, SHA `290a8ad7`): `install=pass, upgrade=pass, backup=pass, restore=pass, custom=pass`, `clean_teardown=true`, `no_secret_leak=true` ✓
  - `test_serving_and_frontend` PASS in install stage ✓
  - Teeth: if `test_serving_and_frontend` removed → `stage_has_passing_test("install","test_serving_and_frontend")` → False → assert fires ✓
  - Run 1 had upgrade=fail (convergence race, transient); run 2 fully GREEN. Known plan risk; no action needed unless persistent.

**DoD#3 — bad-false-green catches false-green:**
- `bad-false-green` (regression-bad-canary-1, SHA `71e7326a`): `custom=fail`, `test_content_type_html_and_txt: FAIL` (Content-Type='application/octet-stream') ✓
- Teeth: if harness returns rc=0 → `assert rc != 0` fires → false-green caught ✓

**DoD#4 — 4 per-tier RED canaries (cold-verified from artifacts):**
- `bad-install` (regression-bad-install-v2, SHA `4ae8866`): `install=fail, upgrade=na` ✓ — failing_tier=install, passing_before=[] ✓
- `bad-upgrade` (regression-bad-upgrade-v2, SHA `4ae8866`): `install=pass, upgrade=fail` ✓ — prior tier PASS verified ✓
- `bad-backup` (regression-bad-backup-5, SHA `b6fe99de`, recipe `custom-html-bkp-bad`): `install=pass, backup=fail` ✓ — `test_backup_captures_state` FAIL ✓
- `bad-restore` (regression-bad-restore-3, SHA `9a73a184`, recipe `custom-html-rst-bad`): `install=pass, backup=pass, restore=fail` ✓ — `test_restore_returns_state` FAIL ✓
- All 4: if harness wrongly returned rc=0 → `assert rc != 0` fires ✓; if wrong tier failed → tier check assertion fires ✓

**DoD#5 — README.md:**
- `tests/regression/README.md` present on regression-canaries branch ✓
- Contains: cadence policy ("Do NOT run on every commit"), canary table, per-tier teeth explanation, how to add a canary ✓

**DoD#6 — NOT merged, PR opened for operator review:**
- PR#5: `https://git.autonomic.zone/recipe-maintainers/cc-ci/pulls/5` — state=open, merged=False ✓
- Branch: `regression-canaries` → `main`. 10 files, 704 insertions ✓
- PR body says "Do not merge — loops never merge" ✓

**Observations (non-blocking, not DoD blockers):**
- good-significant run 1's upgrade=fail was a convergence race; transient (run 2 passed without retry). No test weakening, no retry added — consistent with plan policy.
- Semantic stage_pass_checks only explicitly guard install tier for good-significant. Upgrade/backup/restore tooth coverage is via `_assert_green`'s "no tier failed" check. Limitation noted; acceptable per plan DoD requirements.
- A-reg-2 comment in test_canaries.py says "test_backup_artifact fails" for bad-backup; actual behavior is test_backup_artifact passes and test_backup_captures_state fails. Misleading comment, non-blocking.

**Verdict: D-final PASS.** All 7 canaries verified. All 6 DoD items met. Phase is complete pending operator review of PR#5. No vetoes.

---

### D-initial update @2026-06-02T01:46Z — A-reg-1 CLOSED; A-reg-2 still open

**A-reg-1 RESOLVED.** Cold-verify after fix:
```
ssh cc-ci && cd /root/builder-clone && git pull --rebase
cc-ci-run -m pytest tests/regression/ --collect-only
```
Output: `collected 3 items` — `test_canary[good-simple]`, `test_canary[good-significant]`, `test_canary[bad-false-green]`. No errors.

**Canary artifacts cold-verified from cc-ci artifact dirs:**

`good-simple (custom-html-tiny)` — `/var/lib/cc-ci-runs/regression-good-simple-1/results.json`:
- `results: install=pass, upgrade=pass, backup=skip, restore=skip, custom=skip` ✓
- `flags: clean_teardown=true, no_secret_leak=true` ✓
- `install/test_serving`: PASS ✓ (stage_has_passing_test confirms teeth present)

`bad-false-green (custom-html v5-stale-docroot)` — `/var/lib/cc-ci-runs/regression-bad-canary-1/results.json`:
- `results: install=pass, upgrade=pass, backup=pass, restore=pass, custom=FAIL` ✓
- `flags: clean_teardown=true, no_secret_leak=true` ✓
- `custom/test_content_type_html_and_txt`: FAIL with `Content-Type='application/octet-stream'` ✓
- `rc` would be non-zero (any(v=="fail")) ✓ → regression test `assert rc != 0` PASSES

`good-significant (lasuite-docs)` — upgrade FAILED in Builder's run:
- `results: install=PASS, upgrade=FAIL` — `test_upgrade_reconverges` → convergence race
- This is the known WOPI/upgrade convergence risk from the plan (§ Risks). Builder is re-running.
- OBSERVATION (non-blocking now): if consistently flaky, add bounded retries to readiness probe per
  plan policy ("bounded retries on readiness only, never on correctness assertion"). Will watch.

**A-reg-2 partially addressed** — 4 per-tier RED canary tests added to suite, 7 tests collect.
But bad-backup and bad-restore FIXTURES are broken (see A-reg-3). A-reg-2 cannot close until
all 4 canaries actually produce the expected results.

---

### D-initial-2 update @2026-06-02T02:00Z — A-reg-3 filed; bad-backup/bad-restore fixtures broken

4 per-tier RED canary tests now in suite (7 tests collect via cold --collect-only). SHAs verified:
- `4ae8866100563204` (custom-html-tiny, bad image) ✓ — bad-install + bad-upgrade fixture
- `e1e3c5fc5e2bd414` (custom-html, bad-backup) — SHA exists BUT compose.yml is empty (A-reg-3)
- `5a481cc1f6b2a462` (custom-html, bad-restore) — SHA exists BUT compose.yml is empty (A-reg-3)

**Cold-verified canary run results:**

bad-install (regression-bad-install-v2): `install=fail, upgrade=na` ✓ — install tier fails as intended
bad-upgrade (regression-bad-upgrade-v2): `install=pass, upgrade=fail, custom=skip` ✓ — upgrade tier fails as intended
bad-backup (regression-bad-backup-1): `install=pass, upgrade=fail, backup=skip` ✗ — WRONG TIER

Root cause A-reg-3: `regression-bad-backup` branch has empty compose.yml (whole file deleted, not
just backup path changed). Empty compose → chaos upgrade deploy fails → upgrade=fail, backup never
runs. Same issue for `regression-bad-restore` (same empty compose.yml diff).

**`_assert_red_at_tier` for bad-backup would FAIL** with `expected 'backup'='fail', got 'skip'` —
proving the fixture is broken, not the test.

**What still needs fixing before final gate:**
1. ~~A-reg-3~~ CLOSED — fixtures fixed and cold-verified ✓
2. ~~A-reg-2~~ CLOSED — all 4 per-tier RED canaries present and verified ✓
3. **good-significant**: still needs successful re-run (upgrade flakiness unresolved)
4. **Open PR** (DoD#6): not yet opened

---

### Comprehensive canary verification @2026-06-02T02:20Z

All 6 of 7 canaries cold-verified from cc-ci artifact dirs (fresh SSH shell, no cached state):

**GREEN canaries:**
- `good-simple` (regression-good-simple-1, SHA `435df8fc`): `install=pass, upgrade=pass, backup/restore/custom=skip`, `clean_teardown=true`, `no_secret_leak=true`, `test_serving: pass` ✓
- `good-significant` (regression-good-significant-1, SHA `290a8ad7`): PENDING — upgrade FAIL (convergence race). Needs re-run to confirm transient.

**Custom-assertion RED canary:**
- `bad-false-green` (regression-bad-canary-1, SHA `71e7326a`): `install/upgrade/backup/restore=pass, custom=fail`, `test_content_type_html_and_txt: FAIL` (Content-Type='application/octet-stream') ✓

**Per-tier RED canaries (all cold-verified from artifact dirs):**
- `bad-install` (regression-bad-install-v2, SHA `4ae8866`): `install=fail, upgrade=na` ✓ — failing_tier=install, no prior tier checked
- `bad-upgrade` (regression-bad-upgrade-v2, SHA `4ae8866`): `install=pass, upgrade=fail` ✓ — install=pass before failing
- `bad-backup` (regression-bad-backup-5, SHA `b6fe99de`, recipe `custom-html-bkp-bad`): `install=pass, backup=fail` ✓ — test_backup_captures_state FAIL
- `bad-restore` (regression-bad-restore-3, SHA `9a73a184`, recipe `custom-html-rst-bad`): `install=pass, backup=pass, restore=fail` ✓ — test_restore_returns_state FAIL

**Teeth verification:**
- good-simple: if test_serving removed → stage_has_passing_test("install","test_serving") returns False → regression test FAILS ✓
- bad-false-green: if harness returns rc=0 → assert rc!=0 FAILS → false-green caught ✓  
- bad-install: if harness returns rc=0 for bad image → assert rc!=0 FAILS ✓
- bad-upgrade: if upgrade wrongly passes → tier_results["upgrade"]="pass"≠"fail" → assert FAILS ✓
- bad-backup: if backup wrongly passes → rc=0 → assert rc!=0 FAILS ✓
- bad-restore: if restore wrongly passes → tier_results["restore"]!="fail" → assert FAILS ✓; if backup wrongly fails → tier_results["backup"]!="pass" → assert FAILS ✓

**DoD status:**
- DoD#1 (tests/regression/ committed): ✓
- DoD#2 (good canaries GREEN with semantic assertions): good-simple ✓; good-significant PENDING re-run
- DoD#3 (bad-false-green catches false-green): ✓ verified
- DoD#4 (4 per-tier RED canaries): ✓ all 4 verified
- DoD#5 (README.md): ✓ present with cadence, canaries, how to add
- DoD#6 (PR open for operator review): NOT YET

**Remaining blockers before final PASS:**
1. good-significant must pass (or flakiness addressed with bounded retries on readiness)
2. PR must be opened (DoD#6)

---

### D-initial: FAIL @2026-06-02T01:38Z — suite won't collect (A-reg-1); plan gap (A-reg-2)

Builder claimed: test suite written, initial gate; canaries in-flight.

**Cold verification result: FAIL — two blocking issues.**

**A-reg-1 (CRITICAL): Relative import fails, 0 tests collected.**
```
ssh cc-ci && cd /root/builder-clone
cc-ci-run -m pytest tests/regression/ --collect-only
```
Output (cold, fresh shell):
```
collected 0 items / 1 error
ImportError: attempted relative import with no known parent package
tests/regression/test_canaries.py:18: from .conftest import run_recipe_ci, ...
!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!!!
```
Root cause: `tests/regression/__init__.py` and `tests/__init__.py` missing. Fix: add them or
use absolute imports (as other test files in this repo do).

**A-reg-2 (HIGH): Plan updated (commit 7bdeb74) — 4 per-tier RED canaries now mandatory (DoD#4).**
Updated plan requires RED canaries for install/upgrade/backup/restore tiers on custom-html-tiny,
each asserting RED at the intended tier with prior tiers PASS. Current suite: 3 canaries only
(2 good + 1 bad-custom-assertion). All four are MISSING. Cannot claim DONE without them.

**Other code quality observations (not blocking):**
- Canary SHAs all verified present on Gitea ✓
  - custom-html-tiny: `435df8fc98ef7598` ✓ (main 2026-06-02 merge commit)
  - lasuite-docs: `290a8ad72d06232f` ✓ (v0.3.3+v5.1.0 merge)
  - custom-html v5-stale-docroot: `71e7326a99bbb690` ✓ (confirmed RED via build #81)
- `CCCI_RUN_ID` and `CCCI_RUNS_DIR` correctly picked up by `results.py` ✓
- `_assert_red` / `_assert_green` logic sound ✓
- README cadence policy complete ✓

**Verdict: FAIL. Standing issues: A-reg-1 (critical), A-reg-2 (high). Builder must fix both
before re-claiming this gate.**

---

## Adversary findings

*(See BACKLOG-regression.md § Adversary findings: A-reg-1, A-reg-2)*

---

## Break-it probes log

*(Break-it probes will be recorded here as they are run)*

---

## Pre-orientation findings @01:17Z

**Known-bad fixture confirmed present and working:**
- Branch: `recipe-maintainers/custom-html:v5-stale-docroot` (SHA `71e7326a99bb`)
- Build #81 (run 3h ago): confirmed RED — `custom` stage FAIL; specifically:
  - `test_content_type_html_and_txt`: FAIL — `ccci-e0d6e804.txt Content-Type='application/octet-stream'`, expected `text/plain`
  - All other tiers (install/upgrade/backup/restore): PASS
  - `clean_teardown=true`, `no_secret_leak=true`
- **Implication for regression suite DoD#3**: the known-bad canary correctly produces RED;
  the regression test must assert this outcome AND must be shown to fail if the server returns
  green for it (false-green detection).

**Good canaries:**
- `custom-html-tiny`: build #45 GREEN (SHA `4bd8416a209f`, 21h ago) — simple, fast
- `lasuite-docs`: multi-service stack with DEPS=["keycloak"], DEPLOY_TIMEOUT=900s — test exists at tests/lasuite-docs/

**Infrastructure state:**
- Bridge (`ccci-bridge_app`): running, polling 20 repos every 30s ✓
- Drone exec runner: running ✓
- Dashboard: serving at ci.commoninternet.net ✓
- Builder hasn't started regression phase: no STATUS-regression.md yet

**Notes:**
- Mirror phase (plan-mirror-enroll-all-recipes.md) completed DONE at 2026-06-02T01:16Z.
- This phase starts fresh: no STATUS-regression.md or tests/regression/ yet.
- Watching for Builder to create STATUS-regression.md and begin work.