diff --git a/machine-docs/STATUS-2.md b/machine-docs/STATUS-2.md index c4955b5..9997230 100644 --- a/machine-docs/STATUS-2.md +++ b/machine-docs/STATUS-2.md @@ -92,27 +92,55 @@ DoD P2/P5/P6/P7/P8 broadly satisfied; remaining is P1 coverage of the above + Q5 ## In flight (@2026-05-30T19:3x — VETO-clearing cycle) Standing VETO on DONE (REVIEW-2 @16:22:07Z) requires: ghost + discourse + mumble all run **upgrade-to-latest** green with justified `compose.ccci.yml` overlays. Current cycle: -- **ghost F2-14b** — NOT claimed; blocked on a real recipe/harness backup defect P4 restore exposed. - full4 timed out (→ DEPLOY_TIMEOUT bumped 1200→2400, `4a160f6`). full5/6/7 then ran deploy-count=1, - install/upgrade/backup/custom PASS but **restore FAIL** (`ci_marker` absent post-restore). Root cause - (DECISIONS 2026-05-30 "ghost P4 restore dead-end", proven via restic snapshot inspection): `abra app - backup create` INTERMITTENTLY omits the mysql volume from the snapshot (db service still settling - after the upgrade-tier chaos redeploy that recreates it with the head's new backup labels/mount); - restore of a dump-less snapshot + a non-`pipefail` reimport hook silently loses the seeded row. 3rd - identical fail → root-caused DEFINITIVELY (instrumented full8): the db container cycles mid-dump → - backupbot captures an empty mysql path → restore loses data. Pure race (full8 passed by luck). - **FIX SHIPPED** (`68a7c79`/`16c9241`): harness `BACKUP_VERIFY` hook (recipe-scoped, additive, like - READY_PROBE) — after the backup op the harness runs a recipe probe and RE-RUNS the whole backup (≤3x) - if it failed to capture; ghost's probe checks `/var/lib/mysql/backup.sql.gz` is a valid non-empty gzip. - Read-only; weakens no assertion. **Verifying via `/root/ccci-ghost-full9.log`** (clone 16c9241): expect - RUN SUMMARY deploy-count=1, install/upgrade/backup/restore/custom ALL pass (backup tier may log - `backup-verify FAILED ... re-running backup`). On reliable green → claim Q4.4/F2-14b incl upgrade-to-latest. +- **ghost F2-14b — CLAIMED, awaiting Adversary.** Full lifecycle GREEN incl upgrade-to-latest, with + reliable P4 backup-integrity via the new `BACKUP_VERIFY` harness hook. See `## Gate F2-14b` below. - **discourse Q4.6** — overlay committed `845b86c` (`tests/discourse/{compose.ccci.yml,install_steps.sh}` + recipe_meta `UPGRADE_BASE_VERSION=0.7.0+3.3.1`, `CHAOS_BASE_DEPLOY`, `COMPOSE_FILE`). Run shape is now full `install,upgrade,backup,restore,custom` (upgrade-tier deferral WITHDRAWN per Adversary bdef282). Queued to run after ghost (single-node budget — heavy recipes sequenced). PR head `7a2e0e0`. - mumble F2-14c + plausible Q4.7b still open. +## Gate F2-14b — CLAIMED @2026-05-30T22:10Z (ghost upgrade-to-latest + reliable P4 backup-integrity) +**WHAT.** ghost full lifecycle GREEN incl upgrade-to-latest (base 1.1.1+6-alpine → PR-head `ae43ffe`), +with P4 backup data-integrity now RELIABLE via the new harness `BACKUP_VERIFY` hook + backup retry. +Closes the ghost portion of the standing DONE VETO checklist (ghost passes full suite incl +upgrade-to-latest; overlay justified; P4 non-vacuous). Also satisfies the Adversary's F2-14b verdict +bar (REVIEW-2 @21:34Z + non-vacuity bar @68b2ddd): retry shown to CONVERGE (not infinite-flaky) and the +probe DISCRIMINATES (returned False on a real bad backup, True after re-run). + +**WHERE (inputs).** +- Harness fix: commit `3a612fc` (atop `68a7c79`) — `BACKUP_VERIFY` hook in `runner/run_recipe_ci.py` + (`_perform_op` backup branch + meta allowlist) and ghost probe in `tests/ghost/recipe_meta.py`. +- Overlay: `tests/ghost/compose.ccci.yml` (app+db `healthcheck.start_period: 15m`; grace-only), + provided by `tests/ghost/install_steps.sh`; `CHAOS_BASE_DEPLOY=True`, `COMPOSE_FILE=compose.yml:compose.ccci.yml`, + `DEPLOY_TIMEOUT`/`TIMEOUT`=2400. +- recipe-PR head: `ae43ffe34089cb466d00168a3ad71b813f70103f` (recipe-maintainers/ghost branch + `ci/mysql-backup`; ships literal 15m app start_period + `mysql_backup.sh` backup pre-hook + + `backupbot.restore.post-hook` reimport). +- Run log on cc-ci: `/root/ccci-ghost-full10.log`. + +**HOW (cold re-run).** From a fresh clone at `3a612fc`, on cc-ci: +`RECIPE=ghost REF=ae43ffe34089cb466d00168a3ad71b813f70103f PR=1 SRC=recipe-maintainers/ghost cc-ci-run runner/run_recipe_ci.py` + +**EXPECTED.** RUN SUMMARY: `deploy-count = 1`; install/upgrade/backup/restore/custom ALL `pass`. +- P3 (≥2 real functional): `test_post_roundtrip::test_create_post_roundtrip PASSED`, + `test_content_api::test_content_api_settings_endpoint PASSED`, + `test_admin_redirect::test_ghost_admin_route_is_wired PASSED`. +- P4 NON-VACUOUS: `test_upgrade::test_upgrade_preserves_state PASSED`, + `test_backup::test_backup_captures_state PASSED`, + `test_restore::test_restore_returns_state PASSED` (seeded `ci_marker` survives seed→backup→mutate + (DROP)→restore→assert; this was RED in full5/6/7 WITHOUT reliable capture — non-vacuous). +- Backup tier may log `backup-verify FAILED (attempt N/3) — ... re-running backup` then pass — this IS + the retry-converge + discrimination evidence (full10 logged attempt 1 FAILED → re-ran → backup pass). +- Clean teardown: 0 ghost stacks / volumes / secrets after the run. + +**Non-vacuity / discrimination (Adversary's @68b2ddd bar).** full10 backup tier: the probe returned +False on a genuinely incomplete backup (db cycled mid-dump) → harness re-ran `abra app backup create` +→ probe True → backup tier PASS → restore PASS. So the probe is neither always-True (would be a no-op: +full5/6/7 prove bad backups slip through and restore goes RED without it) nor always-False (would never +converge). It is a read-only `gzip -t && wc -c > 0` check; weakens no assertion (restore still re-reads +the seeded row from the restored snapshot — P4 stays the real gate). + ## In flight (history) **Q4.4 ghost — ✅ Adversary PASS @2026-05-30 (REVIEW-2 `baa7ad8`). DONE.** (See ## Gate Q4.4.) Closes the Adversary's standing ghost §4.3 DONE-blocker. 4th data-loss recipe bug cc-ci caught