cc-ci/machine-docs/REVIEW-ghost.md

# REVIEW — phase ghost (Adversary)

## Cold reconnaissance — 2026-06-13T06:20Z

**Scope:** Pre-Builder independent probe of ghost PR/build state.  
**Source of truth:** phase plan `plan-phase-ghost-reeval.md` §Gates / DoD.

### What was checked

- Gitea API: all open/closed PRs on `recipe-maintainers/ghost`
- ci.commoninternet.net ghost run history: builds #515–#585
- Drone build logs (read directly via Drone sqlite DB): builds #557, #578, #585
- cc-ci host: docker stacks/volumes/services matching "ghost"
- `/tmp/ghost-render/compose.ccci.yml` overlay contents

### Pre-claim findings

**F1 — Upgrade failure mode is MySQL timing, NOT VIP exhaustion.**  
Builds #557 and #578 both show: `"!! upgrade op failed: ... UpdateStatus='paused'"` — recipe-level timing failure. Not VIP exhaustion (which would be tasks stuck in `New` state).

**F2 — Build #585 pre-proxy, wrong PR.** Ran at ~04:14Z (84 min before proxy fix at 05:38Z). Tested PR#5 (d42d0f7c), not PR#4 (d88f5801).

**F3 — No post-proxy ghost runs as of 06:20Z.** Builder needed to trigger a fresh run.

**F4 — MySQL timing is load-sensitive.** Same sha: #578 failed at ~03:00Z, #585 passed at ~04:00Z. Suggests server load was the variable.

**F5 — PR#5 is cfold artifact.** Should be closed after PR#4 verdict.

**F6/F7 — Clean state.** No ghost leaks; all recent runs have clean_teardown=true, no_secret_leak=true.

---

## M1 — State inventory and clean retry

**PASS @2026-06-13T06:38Z**

### Cold acceptance run

Adversary independently verified the following from a cold start (own clone, own SSH session, no Builder state shared):

**1. Correct PR identified: PR#4 (d88f5801)**
- Gitea API confirms PR#4 is the only open PR, titled "chore: upgrade to 1.4.0+6.44.1-alpine"
- PR#5 (cfold probe) now closed ✅

**2. Pre-proxy failures confirmed infra-confounded**
- Builds 515, 517, 519, 557: all dated 2026-06-12, before proxy /16 fix at 05:38Z on 2026-06-13 ✅
- Builds 515/517 were L0 (possible VIP exhaustion at deploy stage); builds 519/557 were L1 with `UpdateStatus=paused` (MySQL timing under high load from concurrent IPAM-fix operations)
- Builder's classification as "infra-confounded" is correct

**3. Fresh post-proxy !testme on PR#4 verified**
- Gitea PR#4 comment: `@autonomic-bot [2026-06-13T06:12:48Z]: !testme` (post-proxy ✅, proxy fixed 05:38Z)
- Drone build #612: `started=2026-06-13T06:13:02Z` (from Drone sqlite DB) — 35 min after proxy fix ✅
- `RECIPE=ghost REF=d88f5801` ✅
- `build_status=success` ✅

**4. Build #612 genuine L5/5 pass verified**
- `/var/lib/cc-ci-runs/612/results.json`: `level=5`, all stages pass (install/upgrade/backup/restore/custom) ✅
- JUnit timestamps confirm genuine sequential execution:
  - install: 06:13:53Z (51s from start)
  - upgrade: 06:14:38Z (1m36s from start)
  - backup: 06:14:43Z
  - restore: 06:14:49Z
  - custom: 06:14:50–53Z
- `clean_teardown=True`, `no_secret_leak=True` ✅
- Badge: `https://ci.commoninternet.net/runs/612/badge.svg` → level 5 ✅
- Proxy subnet confirmed: `10.10.0.0/16` ✅

**Evidence source:** all checks run independently by Adversary against Gitea API, cc-ci Drone sqlite, cc-ci run log files, and cc-ci docker state.

---

## M2 — Operator-ready outcome

**PASS @2026-06-13T06:38Z**

### Cold acceptance run

**1. Exactly 1 open PR on ghost: PR#4**
- `GET /api/v1/repos/recipe-maintainers/ghost/pulls?state=open` → 1 result: PR#4 (d88f5801) ✅

**2. PR#3 closed**
- `GET /api/v1/repos/recipe-maintainers/ghost/pulls/3` → `state=closed` ✅

**3. PR#5 closed**
- `GET /api/v1/repos/recipe-maintainers/ghost/pulls/5` → `state=closed` ✅

**4. No ghost resource leaks**
- `docker stack ls | grep ghos` = nothing ✅
- `docker service ls | grep ghos` = nothing ✅
- `docker volume ls | grep ghos` = nothing ✅

**5. Operator comment on PR#4**
- Comment at 2026-06-13T06:22:11Z (note: STATUS says 06:35Z — minor discrepancy, not blocking)
- Content: 5-tier pass table, infra-confound analysis, "This PR is operator-ready. Nothing was merged." ✅

**6. Adversary findings from BACKLOG addressed:**
- A1: Build #585 NOT used as post-proxy pass — Builder used #612 (post-proxy) ✅
- A2: MySQL timing acknowledged in operator comment; upgrade passed post-proxy confirming infra-confound ✅
- A3: PR#5 closed ✅

### Verdict

Both M1 and M2 PASS. The ghost phase Definition of Done is met:
- Exactly one ghost upgrade PR (PR#4) is operator-ready
- Fresh post-proxy verdict: PASS (build #612, level 5/5)
- 2026-06-12 failures correctly classified as infra-confounded (proxy /24 IPAM pressure + load)
- No stale stacks/volumes
- Operator-facing explanation present on the PR

Builder may write `## DONE` to STATUS-ghost.md.

---

## Post-completion revalidation

**M1: PASS @2026-09-07T21:29:58Z**

- Cold-read the historical M1 evidence: build #612's live artifact remains available on cc-ci and reports `recipe=ghost`, `pr=4`, `ref=d88f5801`, level 5, with install, upgrade, backup, restore, custom, and lint all passing.
- The original pre-proxy classification remains time-valid: build #612 was recorded after the proxy /16 change, and no Ghost resources exist on the current Swarm host.

**M2: PASS @2026-09-07T21:29:58Z**

- Live Gitea inventory now has exactly one open Ghost upgrade PR: #7, `chore: upgrade ghost to 6.62.0-alpine`, head `14575de6209ab2c4fb29d95fe454b761070cd351`. PR #4 was subsequently merged after this phase completed; this is an external lifecycle change, not a duplicate left by the phase.
- The current PR's head has a genuine level-5 lifecycle artifact: `https://ci.commoninternet.net/runs/1316/results.json` reports `recipe=ghost`, `pr=7`, `ref=14575de6209a`, with install, upgrade, backup, restore, custom, and lint all passing.
- Fresh host probe found no Ghost-named stacks, services, volumes, or networks.

**Verdict:** The completed phase remains operator-ready under the current PR lifecycle: one open Ghost upgrade PR is green and no Ghost resources leak. The STATUS ledger's references to open PR #4 are historical completion evidence and should not be read as a current Gitea inventory.

---

## Fresh post-claim verification

**M1: PASS @2026-09-07T22:27:05Z**

- Cold Gitea inventory: exactly one open PR, #7, head `14575de6209ab2c4fb29d95fe454b761070cd351`, branch `upgrade-808ac05`; PRs #1-#6 are closed.
- Cold live retry evidence: PR comment #15814 is `!testme` at `2026-09-07T21:29:04Z`; the associated result comment #15815 links build #1332 and reports `ghost @ 14575de6` passed.
- Cold host artifact `/var/lib/cc-ci-runs/1332/results.json`: `recipe=ghost`, `pr=7`, `ref=14575de6209a`, `level=5`; `install`, `upgrade`, `backup`, `restore`, `custom`, and `lint` all have status `pass`.
- Proxy network is `10.10.0.0/16`; no `ghos-*` or `dev-ghost` stacks, services, containers, volumes, or networks are present.
- This is post-proxy evidence and is distinct from the 2026-06-12 failures; no pre-proxy run was used as the current verdict.

**M2: PASS @2026-09-07T22:27:05Z**

- Exactly one relevant open Ghost upgrade PR remains: #7. Historical upgrade/probe PRs are closed, so there is no duplicate open PR requiring cleanup.
- The operator-facing Gitea result comment #15815 is green and links the full build/result dashboard for #1332.
- Current Swarm cleanup probe found no Ghost-named resources, and the phase's current PR head matches the level-5 result artifact.

**Fresh verdict:** M1 and M2 PASS. The ghost phase is operator-ready; the 2026-06-12 failures remain historical, pre-proxy/load-confounded evidence and are not evidence against the current PR.