From 3859cd7f4029aaa5bdf683d25deb391d82ad14b4 Mon Sep 17 00:00:00 2001 From: autonomic-bot Date: Tue, 2 Jun 2026 01:59:50 +0000 Subject: [PATCH] =?UTF-8?q?review(regression):=20A-reg-3=20=E2=80=94=20bad?= =?UTF-8?q?-backup/bad-restore=20compose.yml=20empty=20(wrong=20tier=20fai?= =?UTF-8?q?ls);=20bad-install/bad-upgrade=20PASS=20cold-verified?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- machine-docs/BACKLOG-regression.md | 36 ++++++++++++++++++++++ machine-docs/BUILDER-INBOX.md | 49 ++++++++++++++++++++++++++++++ machine-docs/REVIEW-regression.md | 33 ++++++++++++++++++-- 3 files changed, 116 insertions(+), 2 deletions(-) create mode 100644 machine-docs/BUILDER-INBOX.md diff --git a/machine-docs/BACKLOG-regression.md b/machine-docs/BACKLOG-regression.md index 043caca..45e7776 100644 --- a/machine-docs/BACKLOG-regression.md +++ b/machine-docs/BACKLOG-regression.md @@ -44,6 +44,42 @@ cc-ci-run -m pytest tests/regression/ --collect-only --- +### A-reg-3 [adversary] OPEN — CRITICAL: bad-backup and bad-restore fixtures broken (empty compose.yml) +**Filed:** 2026-06-02T01:58Z +**Severity:** CRITICAL — both fixtures fail at upgrade instead of their intended tier + +Cold-verified by inspecting `regression-bad-backup` and `regression-bad-restore` branches: +```bash +ssh cc-ci 'cd /root/.abra/recipes/custom-html && git diff origin/main..origin/regression-bad-backup -- compose.yml' +``` +Result: compose.yml is completely empty (entire file deleted, leaving only a blank line). Same +for `regression-bad-restore`. + +**Evidence from run artifacts:** +- `regression-bad-backup-1`: `results: install=pass, upgrade=fail, backup=skip` + - Expected: `install=pass, upgrade=pass, backup=fail` + - Actual: upgrade fails because chaos deploy deploys empty compose → no service → deploy error +- `regression-bad-restore-*`: never ran to completion (same root cause blocks it) + +**Impact on regression test assertions:** +`_assert_red_at_tier` for bad-backup: +- `failing_tier="backup"` → checks `results["backup"]="skip"` → FAIL: "expected 'backup'='fail', got 'skip'" +- Test would FAIL with confusing assertion, not passing as expected + +**Fix:** Recreate both fixture branches with correct compose.yml that: +- bad-backup: keeps full valid nginx service, only changes `backupbot.backup.path` label to `/nonexistent-cc-ci-canary-bad` +- bad-restore: keeps full valid nginx service, changes backup scope to capture a subdir that doesn't contain ci-marker.txt (so restore doesn't recover the marker) + +The compose.yml should be identical to main EXCEPT for the single label/config change. + +**Repro:** `git diff origin/main..origin/regression-bad-backup -- compose.yml` → empty file + +**Adversary closes:** after both fixtures are recreated correctly, runs confirm: +- bad-backup: `install=pass, upgrade=pass, backup=fail` +- bad-restore: `install=pass, upgrade=pass, backup=pass, restore=fail` with `test_restore_returns_state` FAIL + +--- + ### A-reg-2 [adversary] OPEN — Plan gap: 4 per-tier RED canaries required by updated DoD **Filed:** 2026-06-02T01:37Z **Severity:** HIGH — DoD#4 unmet; Builder cannot claim DONE without these diff --git a/machine-docs/BUILDER-INBOX.md b/machine-docs/BUILDER-INBOX.md new file mode 100644 index 0000000..bc8375a --- /dev/null +++ b/machine-docs/BUILDER-INBOX.md @@ -0,0 +1,49 @@ +# BUILDER-INBOX — from Adversary @2026-06-02T02:00Z + +## A-reg-3 CRITICAL: bad-backup and bad-restore fixtures have empty compose.yml + +Both `regression-bad-backup` and `regression-bad-restore` branches in custom-html have +**entirely empty compose.yml** (the whole file was deleted). This breaks the upgrade tier +for both canaries — chaos upgrade deploys an empty compose → no service → upgrade fails. + +Cold verification: +```bash +ssh cc-ci 'cd /root/.abra/recipes/custom-html && git diff origin/main..origin/regression-bad-backup -- compose.yml' +``` +Shows: everything from `version: "3.8"` to the end deleted, only a blank line remains. + +**Run artifact evidence:** +- `regression-bad-backup-1`: `install=pass, upgrade=fail, backup=skip` + - Expected: `install=pass, upgrade=pass, backup=fail` +- `regression-bad-restore`: never reached backup/restore (same root cause) + +**`_assert_red_at_tier` for bad-backup fails**: `results["backup"]="skip"` ≠ "fail" + +## Fix required + +For `regression-bad-backup` on custom-html: compose.yml must be the FULL valid compose +(copy of main), with ONLY this label changed: +```yaml +- "backupbot.backup.path=/usr/share/nginx/html" +``` +→ changed to: +```yaml +- "backupbot.backup.path=/nonexistent-path-cc-ci-canary-bad" +``` + +For `regression-bad-restore` on custom-html: compose.yml must be the FULL valid compose +(copy of main), with the backupbot path pointed at a SUBDIR that doesn't contain ci-marker.txt: +```yaml +- "backupbot.backup.path=/usr/share/nginx/html/.backup-data" +``` +(This dir is either empty or doesn't contain ci-marker.txt → backup succeeds but doesn't +capture the marker → restore doesn't recover it → test_restore_returns_state fails: "mutated" +not "original") + +## Status + +- bad-install: ✓ CONFIRMED working (install=fail, upgrade=na) +- bad-upgrade: ✓ CONFIRMED working (install=pass, upgrade=fail) +- bad-backup: ✗ BROKEN (fix needed) +- bad-restore: ✗ BROKEN (fix needed, blocked on backup fix) +- good-significant: still needs re-run (upgrade flakiness check) diff --git a/machine-docs/REVIEW-regression.md b/machine-docs/REVIEW-regression.md index ea9ba51..677d788 100644 --- a/machine-docs/REVIEW-regression.md +++ b/machine-docs/REVIEW-regression.md @@ -38,8 +38,37 @@ Output: `collected 3 items` — `test_canary[good-simple]`, `test_canary[good-si - OBSERVATION (non-blocking now): if consistently flaky, add bounded retries to readiness probe per plan policy ("bounded retries on readiness only, never on correctness assertion"). Will watch. -**A-reg-2 still open** — 4 per-tier RED canaries (install/upgrade/backup/restore on -custom-html-tiny) not yet implemented. Builder actively working on it. See BACKLOG-regression.md. +**A-reg-2 partially addressed** — 4 per-tier RED canary tests added to suite, 7 tests collect. +But bad-backup and bad-restore FIXTURES are broken (see A-reg-3). A-reg-2 cannot close until +all 4 canaries actually produce the expected results. + +--- + +### D-initial-2 update @2026-06-02T02:00Z — A-reg-3 filed; bad-backup/bad-restore fixtures broken + +4 per-tier RED canary tests now in suite (7 tests collect via cold --collect-only). SHAs verified: +- `4ae8866100563204` (custom-html-tiny, bad image) ✓ — bad-install + bad-upgrade fixture +- `e1e3c5fc5e2bd414` (custom-html, bad-backup) — SHA exists BUT compose.yml is empty (A-reg-3) +- `5a481cc1f6b2a462` (custom-html, bad-restore) — SHA exists BUT compose.yml is empty (A-reg-3) + +**Cold-verified canary run results:** + +bad-install (regression-bad-install-v2): `install=fail, upgrade=na` ✓ — install tier fails as intended +bad-upgrade (regression-bad-upgrade-v2): `install=pass, upgrade=fail, custom=skip` ✓ — upgrade tier fails as intended +bad-backup (regression-bad-backup-1): `install=pass, upgrade=fail, backup=skip` ✗ — WRONG TIER + +Root cause A-reg-3: `regression-bad-backup` branch has empty compose.yml (whole file deleted, not +just backup path changed). Empty compose → chaos upgrade deploy fails → upgrade=fail, backup never +runs. Same issue for `regression-bad-restore` (same empty compose.yml diff). + +**`_assert_red_at_tier` for bad-backup would FAIL** with `expected 'backup'='fail', got 'skip'` — +proving the fixture is broken, not the test. + +**What still needs fixing before final gate:** +1. A-reg-3: Recreate bad-backup and bad-restore fixtures with correct compose.yml (only targeted change) +2. Run bad-backup and bad-restore to confirm correct tier failures +3. Re-run good-significant (lasuite-docs) to confirm upgrade race is transient +4. Open PR ---