claim(2): F2-14b ghost — full lifecycle GREEN incl upgrade-to-latest + reliable P4 (BACKUP_VERIFY)
full10 (/root/ccci-ghost-full10.log, clone 3a612fc): deploy-count=1; install/upgrade/backup/restore/
custom ALL pass. P3: create-post + content-api + admin-redirect PASSED. P4 non-vacuous: upgrade/backup/
restore state PASSED (ci_marker survives seed→backup→mutate→restore — RED in full5/6/7 pre-fix). The
backup-verify retry CONVERGED + DISCRIMINATED in-situ (attempt 1 FAILED on a real bad backup → re-ran →
pass). Clean teardown (0/0/0). Verify per ## Gate F2-14b in STATUS-2.
This commit is contained in:
@ -92,27 +92,55 @@ DoD P2/P5/P6/P7/P8 broadly satisfied; remaining is P1 coverage of the above + Q5
|
||||
## In flight (@2026-05-30T19:3x — VETO-clearing cycle)
|
||||
Standing VETO on DONE (REVIEW-2 @16:22:07Z) requires: ghost + discourse + mumble all run
|
||||
**upgrade-to-latest** green with justified `compose.ccci.yml` overlays. Current cycle:
|
||||
- **ghost F2-14b** — NOT claimed; blocked on a real recipe/harness backup defect P4 restore exposed.
|
||||
full4 timed out (→ DEPLOY_TIMEOUT bumped 1200→2400, `4a160f6`). full5/6/7 then ran deploy-count=1,
|
||||
install/upgrade/backup/custom PASS but **restore FAIL** (`ci_marker` absent post-restore). Root cause
|
||||
(DECISIONS 2026-05-30 "ghost P4 restore dead-end", proven via restic snapshot inspection): `abra app
|
||||
backup create` INTERMITTENTLY omits the mysql volume from the snapshot (db service still settling
|
||||
after the upgrade-tier chaos redeploy that recreates it with the head's new backup labels/mount);
|
||||
restore of a dump-less snapshot + a non-`pipefail` reimport hook silently loses the seeded row. 3rd
|
||||
identical fail → root-caused DEFINITIVELY (instrumented full8): the db container cycles mid-dump →
|
||||
backupbot captures an empty mysql path → restore loses data. Pure race (full8 passed by luck).
|
||||
**FIX SHIPPED** (`68a7c79`/`16c9241`): harness `BACKUP_VERIFY` hook (recipe-scoped, additive, like
|
||||
READY_PROBE) — after the backup op the harness runs a recipe probe and RE-RUNS the whole backup (≤3x)
|
||||
if it failed to capture; ghost's probe checks `/var/lib/mysql/backup.sql.gz` is a valid non-empty gzip.
|
||||
Read-only; weakens no assertion. **Verifying via `/root/ccci-ghost-full9.log`** (clone 16c9241): expect
|
||||
RUN SUMMARY deploy-count=1, install/upgrade/backup/restore/custom ALL pass (backup tier may log
|
||||
`backup-verify FAILED ... re-running backup`). On reliable green → claim Q4.4/F2-14b incl upgrade-to-latest.
|
||||
- **ghost F2-14b — CLAIMED, awaiting Adversary.** Full lifecycle GREEN incl upgrade-to-latest, with
|
||||
reliable P4 backup-integrity via the new `BACKUP_VERIFY` harness hook. See `## Gate F2-14b` below.
|
||||
- **discourse Q4.6** — overlay committed `845b86c` (`tests/discourse/{compose.ccci.yml,install_steps.sh}`
|
||||
+ recipe_meta `UPGRADE_BASE_VERSION=0.7.0+3.3.1`, `CHAOS_BASE_DEPLOY`, `COMPOSE_FILE`). Run shape is
|
||||
now full `install,upgrade,backup,restore,custom` (upgrade-tier deferral WITHDRAWN per Adversary
|
||||
bdef282). Queued to run after ghost (single-node budget — heavy recipes sequenced). PR head `7a2e0e0`.
|
||||
- mumble F2-14c + plausible Q4.7b still open.
|
||||
|
||||
## Gate F2-14b — CLAIMED @2026-05-30T22:10Z (ghost upgrade-to-latest + reliable P4 backup-integrity)
|
||||
**WHAT.** ghost full lifecycle GREEN incl upgrade-to-latest (base 1.1.1+6-alpine → PR-head `ae43ffe`),
|
||||
with P4 backup data-integrity now RELIABLE via the new harness `BACKUP_VERIFY` hook + backup retry.
|
||||
Closes the ghost portion of the standing DONE VETO checklist (ghost passes full suite incl
|
||||
upgrade-to-latest; overlay justified; P4 non-vacuous). Also satisfies the Adversary's F2-14b verdict
|
||||
bar (REVIEW-2 @21:34Z + non-vacuity bar @68b2ddd): retry shown to CONVERGE (not infinite-flaky) and the
|
||||
probe DISCRIMINATES (returned False on a real bad backup, True after re-run).
|
||||
|
||||
**WHERE (inputs).**
|
||||
- Harness fix: commit `3a612fc` (atop `68a7c79`) — `BACKUP_VERIFY` hook in `runner/run_recipe_ci.py`
|
||||
(`_perform_op` backup branch + meta allowlist) and ghost probe in `tests/ghost/recipe_meta.py`.
|
||||
- Overlay: `tests/ghost/compose.ccci.yml` (app+db `healthcheck.start_period: 15m`; grace-only),
|
||||
provided by `tests/ghost/install_steps.sh`; `CHAOS_BASE_DEPLOY=True`, `COMPOSE_FILE=compose.yml:compose.ccci.yml`,
|
||||
`DEPLOY_TIMEOUT`/`TIMEOUT`=2400.
|
||||
- recipe-PR head: `ae43ffe34089cb466d00168a3ad71b813f70103f` (recipe-maintainers/ghost branch
|
||||
`ci/mysql-backup`; ships literal 15m app start_period + `mysql_backup.sh` backup pre-hook +
|
||||
`backupbot.restore.post-hook` reimport).
|
||||
- Run log on cc-ci: `/root/ccci-ghost-full10.log`.
|
||||
|
||||
**HOW (cold re-run).** From a fresh clone at `3a612fc`, on cc-ci:
|
||||
`RECIPE=ghost REF=ae43ffe34089cb466d00168a3ad71b813f70103f PR=1 SRC=recipe-maintainers/ghost cc-ci-run runner/run_recipe_ci.py`
|
||||
|
||||
**EXPECTED.** RUN SUMMARY: `deploy-count = 1`; install/upgrade/backup/restore/custom ALL `pass`.
|
||||
- P3 (≥2 real functional): `test_post_roundtrip::test_create_post_roundtrip PASSED`,
|
||||
`test_content_api::test_content_api_settings_endpoint PASSED`,
|
||||
`test_admin_redirect::test_ghost_admin_route_is_wired PASSED`.
|
||||
- P4 NON-VACUOUS: `test_upgrade::test_upgrade_preserves_state PASSED`,
|
||||
`test_backup::test_backup_captures_state PASSED`,
|
||||
`test_restore::test_restore_returns_state PASSED` (seeded `ci_marker` survives seed→backup→mutate
|
||||
(DROP)→restore→assert; this was RED in full5/6/7 WITHOUT reliable capture — non-vacuous).
|
||||
- Backup tier may log `backup-verify FAILED (attempt N/3) — ... re-running backup` then pass — this IS
|
||||
the retry-converge + discrimination evidence (full10 logged attempt 1 FAILED → re-ran → backup pass).
|
||||
- Clean teardown: 0 ghost stacks / volumes / secrets after the run.
|
||||
|
||||
**Non-vacuity / discrimination (Adversary's @68b2ddd bar).** full10 backup tier: the probe returned
|
||||
False on a genuinely incomplete backup (db cycled mid-dump) → harness re-ran `abra app backup create`
|
||||
→ probe True → backup tier PASS → restore PASS. So the probe is neither always-True (would be a no-op:
|
||||
full5/6/7 prove bad backups slip through and restore goes RED without it) nor always-False (would never
|
||||
converge). It is a read-only `gzip -t && wc -c > 0` check; weakens no assertion (restore still re-reads
|
||||
the seeded row from the restored snapshot — P4 stays the real gate).
|
||||
|
||||
## In flight (history)
|
||||
**Q4.4 ghost — ✅ Adversary PASS @2026-05-30 (REVIEW-2 `baa7ad8`). DONE.** (See ## Gate Q4.4.)
|
||||
Closes the Adversary's standing ghost §4.3 DONE-blocker. 4th data-loss recipe bug cc-ci caught
|
||||
|
||||
Reference in New Issue
Block a user