review(regression): A-reg-3 — bad-backup/bad-restore compose.yml empty (wrong tier fails); bad-install/bad-upgrade PASS cold-verified
Some checks failed
continuous-integration/drone/push Build is failing
Some checks failed
continuous-integration/drone/push Build is failing
This commit is contained in:
autonomic-bot
@ -44,6 +44,42 @@ cc-ci-run -m pytest tests/regression/ --collect-only
|
||||
|
||||
---
|
||||
|
||||
### A-reg-3 [adversary] OPEN — CRITICAL: bad-backup and bad-restore fixtures broken (empty compose.yml)
|
||||
**Filed:** 2026-06-02T01:58Z
|
||||
**Severity:** CRITICAL — both fixtures fail at upgrade instead of their intended tier
|
||||
|
||||
Cold-verified by inspecting `regression-bad-backup` and `regression-bad-restore` branches:
|
||||
```bash
|
||||
ssh cc-ci 'cd /root/.abra/recipes/custom-html && git diff origin/main..origin/regression-bad-backup -- compose.yml'
|
||||
```
|
||||
Result: compose.yml is completely empty (entire file deleted, leaving only a blank line). Same
|
||||
for `regression-bad-restore`.
|
||||
|
||||
**Evidence from run artifacts:**
|
||||
- `regression-bad-backup-1`: `results: install=pass, upgrade=fail, backup=skip`
|
||||
- Expected: `install=pass, upgrade=pass, backup=fail`
|
||||
- Actual: upgrade fails because chaos deploy deploys empty compose → no service → deploy error
|
||||
- `regression-bad-restore-*`: never ran to completion (same root cause blocks it)
|
||||
|
||||
**Impact on regression test assertions:**
|
||||
`_assert_red_at_tier` for bad-backup:
|
||||
- `failing_tier="backup"` → checks `results["backup"]="skip"` → FAIL: "expected 'backup'='fail', got 'skip'"
|
||||
- Test would FAIL with confusing assertion, not passing as expected
|
||||
|
||||
**Fix:** Recreate both fixture branches with correct compose.yml that:
|
||||
- bad-backup: keeps full valid nginx service, only changes `backupbot.backup.path` label to `/nonexistent-cc-ci-canary-bad`
|
||||
- bad-restore: keeps full valid nginx service, changes backup scope to capture a subdir that doesn't contain ci-marker.txt (so restore doesn't recover the marker)
|
||||
|
||||
The compose.yml should be identical to main EXCEPT for the single label/config change.
|
||||
|
||||
**Repro:** `git diff origin/main..origin/regression-bad-backup -- compose.yml` → empty file
|
||||
|
||||
**Adversary closes:** after both fixtures are recreated correctly, runs confirm:
|
||||
- bad-backup: `install=pass, upgrade=pass, backup=fail`
|
||||
- bad-restore: `install=pass, upgrade=pass, backup=pass, restore=fail` with `test_restore_returns_state` FAIL
|
||||
|
||||
---
|
||||
|
||||
### A-reg-2 [adversary] OPEN — Plan gap: 4 per-tier RED canaries required by updated DoD
|
||||
**Filed:** 2026-06-02T01:37Z
|
||||
**Severity:** HIGH — DoD#4 unmet; Builder cannot claim DONE without these
|
||||
|
||||
49
machine-docs/BUILDER-INBOX.md
Normal file
49
machine-docs/BUILDER-INBOX.md
Normal file
@ -0,0 +1,49 @@
|
||||
# BUILDER-INBOX — from Adversary @2026-06-02T02:00Z
|
||||
|
||||
## A-reg-3 CRITICAL: bad-backup and bad-restore fixtures have empty compose.yml
|
||||
|
||||
Both `regression-bad-backup` and `regression-bad-restore` branches in custom-html have
|
||||
**entirely empty compose.yml** (the whole file was deleted). This breaks the upgrade tier
|
||||
for both canaries — chaos upgrade deploys an empty compose → no service → upgrade fails.
|
||||
|
||||
Cold verification:
|
||||
```bash
|
||||
ssh cc-ci 'cd /root/.abra/recipes/custom-html && git diff origin/main..origin/regression-bad-backup -- compose.yml'
|
||||
```
|
||||
Shows: everything from `version: "3.8"` to the end deleted, only a blank line remains.
|
||||
|
||||
**Run artifact evidence:**
|
||||
- `regression-bad-backup-1`: `install=pass, upgrade=fail, backup=skip`
|
||||
- Expected: `install=pass, upgrade=pass, backup=fail`
|
||||
- `regression-bad-restore`: never reached backup/restore (same root cause)
|
||||
|
||||
**`_assert_red_at_tier` for bad-backup fails**: `results["backup"]="skip"` ≠ "fail"
|
||||
|
||||
## Fix required
|
||||
|
||||
For `regression-bad-backup` on custom-html: compose.yml must be the FULL valid compose
|
||||
(copy of main), with ONLY this label changed:
|
||||
```yaml
|
||||
- "backupbot.backup.path=/usr/share/nginx/html"
|
||||
```
|
||||
→ changed to:
|
||||
```yaml
|
||||
- "backupbot.backup.path=/nonexistent-path-cc-ci-canary-bad"
|
||||
```
|
||||
|
||||
For `regression-bad-restore` on custom-html: compose.yml must be the FULL valid compose
|
||||
(copy of main), with the backupbot path pointed at a SUBDIR that doesn't contain ci-marker.txt:
|
||||
```yaml
|
||||
- "backupbot.backup.path=/usr/share/nginx/html/.backup-data"
|
||||
```
|
||||
(This dir is either empty or doesn't contain ci-marker.txt → backup succeeds but doesn't
|
||||
capture the marker → restore doesn't recover it → test_restore_returns_state fails: "mutated"
|
||||
not "original")
|
||||
|
||||
## Status
|
||||
|
||||
- bad-install: ✓ CONFIRMED working (install=fail, upgrade=na)
|
||||
- bad-upgrade: ✓ CONFIRMED working (install=pass, upgrade=fail)
|
||||
- bad-backup: ✗ BROKEN (fix needed)
|
||||
- bad-restore: ✗ BROKEN (fix needed, blocked on backup fix)
|
||||
- good-significant: still needs re-run (upgrade flakiness check)
|
||||
@ -38,8 +38,37 @@ Output: `collected 3 items` — `test_canary[good-simple]`, `test_canary[good-si
|
||||
- OBSERVATION (non-blocking now): if consistently flaky, add bounded retries to readiness probe per
|
||||
plan policy ("bounded retries on readiness only, never on correctness assertion"). Will watch.
|
||||
|
||||
**A-reg-2 still open** — 4 per-tier RED canaries (install/upgrade/backup/restore on
|
||||
custom-html-tiny) not yet implemented. Builder actively working on it. See BACKLOG-regression.md.
|
||||
**A-reg-2 partially addressed** — 4 per-tier RED canary tests added to suite, 7 tests collect.
|
||||
But bad-backup and bad-restore FIXTURES are broken (see A-reg-3). A-reg-2 cannot close until
|
||||
all 4 canaries actually produce the expected results.
|
||||
|
||||
---
|
||||
|
||||
### D-initial-2 update @2026-06-02T02:00Z — A-reg-3 filed; bad-backup/bad-restore fixtures broken
|
||||
|
||||
4 per-tier RED canary tests now in suite (7 tests collect via cold --collect-only). SHAs verified:
|
||||
- `4ae8866100563204` (custom-html-tiny, bad image) ✓ — bad-install + bad-upgrade fixture
|
||||
- `e1e3c5fc5e2bd414` (custom-html, bad-backup) — SHA exists BUT compose.yml is empty (A-reg-3)
|
||||
- `5a481cc1f6b2a462` (custom-html, bad-restore) — SHA exists BUT compose.yml is empty (A-reg-3)
|
||||
|
||||
**Cold-verified canary run results:**
|
||||
|
||||
bad-install (regression-bad-install-v2): `install=fail, upgrade=na` ✓ — install tier fails as intended
|
||||
bad-upgrade (regression-bad-upgrade-v2): `install=pass, upgrade=fail, custom=skip` ✓ — upgrade tier fails as intended
|
||||
bad-backup (regression-bad-backup-1): `install=pass, upgrade=fail, backup=skip` ✗ — WRONG TIER
|
||||
|
||||
Root cause A-reg-3: `regression-bad-backup` branch has empty compose.yml (whole file deleted, not
|
||||
just backup path changed). Empty compose → chaos upgrade deploy fails → upgrade=fail, backup never
|
||||
runs. Same issue for `regression-bad-restore` (same empty compose.yml diff).
|
||||
|
||||
**`_assert_red_at_tier` for bad-backup would FAIL** with `expected 'backup'='fail', got 'skip'` —
|
||||
proving the fixture is broken, not the test.
|
||||
|
||||
**What still needs fixing before final gate:**
|
||||
1. A-reg-3: Recreate bad-backup and bad-restore fixtures with correct compose.yml (only targeted change)
|
||||
2. Run bad-backup and bad-restore to confirm correct tier failures
|
||||
3. Re-run good-significant (lasuite-docs) to confirm upgrade race is transient
|
||||
4. Open PR
|
||||
|
||||
---
|
||||
|
||||
|
||||
Reference in New Issue
Block a user