claim(2): F2-14b ghost — full lifecycle GREEN incl upgrade-to-latest + reliable P4 (BACKUP_VERIFY)

full10 (/root/ccci-ghost-full10.log, clone 3a612fc): deploy-count=1; install/upgrade/backup/restore/
custom ALL pass. P3: create-post + content-api + admin-redirect PASSED. P4 non-vacuous: upgrade/backup/
restore state PASSED (ci_marker survives seed→backup→mutate→restore — RED in full5/6/7 pre-fix). The
backup-verify retry CONVERGED + DISCRIMINATED in-situ (attempt 1 FAILED on a real bad backup → re-ran →
pass). Clean teardown (0/0/0). Verify per ## Gate F2-14b in STATUS-2.
This commit is contained in:
autonomic-bot
2026-05-30 22:13:20 +00:00
parent 68b2dddf42
commit be0475ae09

View File

@ -92,27 +92,55 @@ DoD P2/P5/P6/P7/P8 broadly satisfied; remaining is P1 coverage of the above + Q5
## In flight (@2026-05-30T19:3x — VETO-clearing cycle)
Standing VETO on DONE (REVIEW-2 @16:22:07Z) requires: ghost + discourse + mumble all run
**upgrade-to-latest** green with justified `compose.ccci.yml` overlays. Current cycle:
- **ghost F2-14b**NOT claimed; blocked on a real recipe/harness backup defect P4 restore exposed.
full4 timed out (→ DEPLOY_TIMEOUT bumped 1200→2400, `4a160f6`). full5/6/7 then ran deploy-count=1,
install/upgrade/backup/custom PASS but **restore FAIL** (`ci_marker` absent post-restore). Root cause
(DECISIONS 2026-05-30 "ghost P4 restore dead-end", proven via restic snapshot inspection): `abra app
backup create` INTERMITTENTLY omits the mysql volume from the snapshot (db service still settling
after the upgrade-tier chaos redeploy that recreates it with the head's new backup labels/mount);
restore of a dump-less snapshot + a non-`pipefail` reimport hook silently loses the seeded row. 3rd
identical fail → root-caused DEFINITIVELY (instrumented full8): the db container cycles mid-dump →
backupbot captures an empty mysql path → restore loses data. Pure race (full8 passed by luck).
**FIX SHIPPED** (`68a7c79`/`16c9241`): harness `BACKUP_VERIFY` hook (recipe-scoped, additive, like
READY_PROBE) — after the backup op the harness runs a recipe probe and RE-RUNS the whole backup (≤3x)
if it failed to capture; ghost's probe checks `/var/lib/mysql/backup.sql.gz` is a valid non-empty gzip.
Read-only; weakens no assertion. **Verifying via `/root/ccci-ghost-full9.log`** (clone 16c9241): expect
RUN SUMMARY deploy-count=1, install/upgrade/backup/restore/custom ALL pass (backup tier may log
`backup-verify FAILED ... re-running backup`). On reliable green → claim Q4.4/F2-14b incl upgrade-to-latest.
- **ghost F2-14b — CLAIMED, awaiting Adversary.** Full lifecycle GREEN incl upgrade-to-latest, with
reliable P4 backup-integrity via the new `BACKUP_VERIFY` harness hook. See `## Gate F2-14b` below.
- **discourse Q4.6** — overlay committed `845b86c` (`tests/discourse/{compose.ccci.yml,install_steps.sh}`
+ recipe_meta `UPGRADE_BASE_VERSION=0.7.0+3.3.1`, `CHAOS_BASE_DEPLOY`, `COMPOSE_FILE`). Run shape is
now full `install,upgrade,backup,restore,custom` (upgrade-tier deferral WITHDRAWN per Adversary
bdef282). Queued to run after ghost (single-node budget — heavy recipes sequenced). PR head `7a2e0e0`.
- mumble F2-14c + plausible Q4.7b still open.
## Gate F2-14b — CLAIMED @2026-05-30T22:10Z (ghost upgrade-to-latest + reliable P4 backup-integrity)
**WHAT.** ghost full lifecycle GREEN incl upgrade-to-latest (base 1.1.1+6-alpine → PR-head `ae43ffe`),
with P4 backup data-integrity now RELIABLE via the new harness `BACKUP_VERIFY` hook + backup retry.
Closes the ghost portion of the standing DONE VETO checklist (ghost passes full suite incl
upgrade-to-latest; overlay justified; P4 non-vacuous). Also satisfies the Adversary's F2-14b verdict
bar (REVIEW-2 @21:34Z + non-vacuity bar @68b2ddd): retry shown to CONVERGE (not infinite-flaky) and the
probe DISCRIMINATES (returned False on a real bad backup, True after re-run).
**WHERE (inputs).**
- Harness fix: commit `3a612fc` (atop `68a7c79`) — `BACKUP_VERIFY` hook in `runner/run_recipe_ci.py`
(`_perform_op` backup branch + meta allowlist) and ghost probe in `tests/ghost/recipe_meta.py`.
- Overlay: `tests/ghost/compose.ccci.yml` (app+db `healthcheck.start_period: 15m`; grace-only),
provided by `tests/ghost/install_steps.sh`; `CHAOS_BASE_DEPLOY=True`, `COMPOSE_FILE=compose.yml:compose.ccci.yml`,
`DEPLOY_TIMEOUT`/`TIMEOUT`=2400.
- recipe-PR head: `ae43ffe34089cb466d00168a3ad71b813f70103f` (recipe-maintainers/ghost branch
`ci/mysql-backup`; ships literal 15m app start_period + `mysql_backup.sh` backup pre-hook +
`backupbot.restore.post-hook` reimport).
- Run log on cc-ci: `/root/ccci-ghost-full10.log`.
**HOW (cold re-run).** From a fresh clone at `3a612fc`, on cc-ci:
`RECIPE=ghost REF=ae43ffe34089cb466d00168a3ad71b813f70103f PR=1 SRC=recipe-maintainers/ghost cc-ci-run runner/run_recipe_ci.py`
**EXPECTED.** RUN SUMMARY: `deploy-count = 1`; install/upgrade/backup/restore/custom ALL `pass`.
- P3 (≥2 real functional): `test_post_roundtrip::test_create_post_roundtrip PASSED`,
`test_content_api::test_content_api_settings_endpoint PASSED`,
`test_admin_redirect::test_ghost_admin_route_is_wired PASSED`.
- P4 NON-VACUOUS: `test_upgrade::test_upgrade_preserves_state PASSED`,
`test_backup::test_backup_captures_state PASSED`,
`test_restore::test_restore_returns_state PASSED` (seeded `ci_marker` survives seed→backup→mutate
(DROP)→restore→assert; this was RED in full5/6/7 WITHOUT reliable capture — non-vacuous).
- Backup tier may log `backup-verify FAILED (attempt N/3) — ... re-running backup` then pass — this IS
the retry-converge + discrimination evidence (full10 logged attempt 1 FAILED → re-ran → backup pass).
- Clean teardown: 0 ghost stacks / volumes / secrets after the run.
**Non-vacuity / discrimination (Adversary's @68b2ddd bar).** full10 backup tier: the probe returned
False on a genuinely incomplete backup (db cycled mid-dump) → harness re-ran `abra app backup create`
→ probe True → backup tier PASS → restore PASS. So the probe is neither always-True (would be a no-op:
full5/6/7 prove bad backups slip through and restore goes RED without it) nor always-False (would never
converge). It is a read-only `gzip -t && wc -c > 0` check; weakens no assertion (restore still re-reads
the seeded row from the restored snapshot — P4 stays the real gate).
## In flight (history)
**Q4.4 ghost — ✅ Adversary PASS @2026-05-30 (REVIEW-2 `baa7ad8`). DONE.** (See ## Gate Q4.4.)
Closes the Adversary's standing ghost §4.3 DONE-blocker. 4th data-loss recipe bug cc-ci caught