cc-ci/machine-docs/REVIEW-ghost.md

# REVIEW — phase ghost (Adversary)

## Cold reconnaissance — 2026-06-13T06:20Z

**Scope:** Pre-Builder independent probe of ghost PR/build state.  
**Source of truth:** phase plan `plan-phase-ghost-reeval.md` §Gates / DoD.

### What was checked

- Gitea API: all open/closed PRs on `recipe-maintainers/ghost`
- ci.commoninternet.net ghost run history: builds #515–#585
- Drone build logs (read directly via Drone sqlite DB): builds #557, #578, #585
- cc-ci host: docker stacks/volumes/services matching "ghost"
- `/tmp/ghost-render/compose.ccci.yml` overlay contents

### Pre-claim findings

**F1 — Upgrade failure mode is MySQL timing, NOT VIP exhaustion.**  
Builds #557 and #578 both show: `"!! upgrade op failed: ... UpdateStatus='paused'"` — recipe-level timing failure. Not VIP exhaustion (which would be tasks stuck in `New` state).

**F2 — Build #585 pre-proxy, wrong PR.** Ran at ~04:14Z (84 min before proxy fix at 05:38Z). Tested PR#5 (d42d0f7c), not PR#4 (d88f5801).

**F3 — No post-proxy ghost runs as of 06:20Z.** Builder needed to trigger a fresh run.

**F4 — MySQL timing is load-sensitive.** Same sha: #578 failed at ~03:00Z, #585 passed at ~04:00Z. Suggests server load was the variable.

**F5 — PR#5 is cfold artifact.** Should be closed after PR#4 verdict.

**F6/F7 — Clean state.** No ghost leaks; all recent runs have clean_teardown=true, no_secret_leak=true.

---

## M1 — State inventory and clean retry

**PASS @2026-06-13T06:38Z**

### Cold acceptance run

Adversary independently verified the following from a cold start (own clone, own SSH session, no Builder state shared):

**1. Correct PR identified: PR#4 (d88f5801)**
- Gitea API confirms PR#4 is the only open PR, titled "chore: upgrade to 1.4.0+6.44.1-alpine"
- PR#5 (cfold probe) now closed ✅

**2. Pre-proxy failures confirmed infra-confounded**
- Builds 515, 517, 519, 557: all dated 2026-06-12, before proxy /16 fix at 05:38Z on 2026-06-13 ✅
- Builds 515/517 were L0 (possible VIP exhaustion at deploy stage); builds 519/557 were L1 with `UpdateStatus=paused` (MySQL timing under high load from concurrent IPAM-fix operations)
- Builder's classification as "infra-confounded" is correct

**3. Fresh post-proxy !testme on PR#4 verified**
- Gitea PR#4 comment: `@autonomic-bot [2026-06-13T06:12:48Z]: !testme` (post-proxy ✅, proxy fixed 05:38Z)
- Drone build #612: `started=2026-06-13T06:13:02Z` (from Drone sqlite DB) — 35 min after proxy fix ✅
- `RECIPE=ghost REF=d88f5801` ✅
- `build_status=success` ✅

**4. Build #612 genuine L5/5 pass verified**
- `/var/lib/cc-ci-runs/612/results.json`: `level=5`, all stages pass (install/upgrade/backup/restore/custom) ✅
- JUnit timestamps confirm genuine sequential execution:
  - install: 06:13:53Z (51s from start)
  - upgrade: 06:14:38Z (1m36s from start)
  - backup: 06:14:43Z
  - restore: 06:14:49Z
  - custom: 06:14:50–53Z
- `clean_teardown=True`, `no_secret_leak=True` ✅
- Badge: `https://ci.commoninternet.net/runs/612/badge.svg` → level 5 ✅
- Proxy subnet confirmed: `10.10.0.0/16` ✅

**Evidence source:** all checks run independently by Adversary against Gitea API, cc-ci Drone sqlite, cc-ci run log files, and cc-ci docker state.

---

## M2 — Operator-ready outcome

**PASS @2026-06-13T06:38Z**

### Cold acceptance run

**1. Exactly 1 open PR on ghost: PR#4**
- `GET /api/v1/repos/recipe-maintainers/ghost/pulls?state=open` → 1 result: PR#4 (d88f5801) ✅

**2. PR#3 closed**
- `GET /api/v1/repos/recipe-maintainers/ghost/pulls/3` → `state=closed` ✅

**3. PR#5 closed**
- `GET /api/v1/repos/recipe-maintainers/ghost/pulls/5` → `state=closed` ✅

**4. No ghost resource leaks**
- `docker stack ls | grep ghos` = nothing ✅
- `docker service ls | grep ghos` = nothing ✅
- `docker volume ls | grep ghos` = nothing ✅

**5. Operator comment on PR#4**
- Comment at 2026-06-13T06:22:11Z (note: STATUS says 06:35Z — minor discrepancy, not blocking)
- Content: 5-tier pass table, infra-confound analysis, "This PR is operator-ready. Nothing was merged." ✅

**6. Adversary findings from BACKLOG addressed:**
- A1: Build #585 NOT used as post-proxy pass — Builder used #612 (post-proxy) ✅
- A2: MySQL timing acknowledged in operator comment; upgrade passed post-proxy confirming infra-confound ✅
- A3: PR#5 closed ✅

### Verdict

Both M1 and M2 PASS. The ghost phase Definition of Done is met:
- Exactly one ghost upgrade PR (PR#4) is operator-ready
- Fresh post-proxy verdict: PASS (build #612, level 5/5)
- 2026-06-12 failures correctly classified as infra-confounded (proxy /24 IPAM pressure + load)
- No stale stacks/volumes
- Operator-facing explanation present on the PR

Builder may write `## DONE` to STATUS-ghost.md.

---

## Post-completion revalidation

**M1: PASS @2026-09-07T21:29:58Z**

- Cold-read the historical M1 evidence: build #612's live artifact remains available on cc-ci and reports `recipe=ghost`, `pr=4`, `ref=d88f5801`, level 5, with install, upgrade, backup, restore, custom, and lint all passing.
- The original pre-proxy classification remains time-valid: build #612 was recorded after the proxy /16 change, and no Ghost resources exist on the current Swarm host.

**M2: PASS @2026-09-07T21:29:58Z**

- Live Gitea inventory now has exactly one open Ghost upgrade PR: #7, `chore: upgrade ghost to 6.62.0-alpine`, head `14575de6209ab2c4fb29d95fe454b761070cd351`. PR #4 was subsequently merged after this phase completed; this is an external lifecycle change, not a duplicate left by the phase.
- The current PR's head has a genuine level-5 lifecycle artifact: `https://ci.commoninternet.net/runs/1316/results.json` reports `recipe=ghost`, `pr=7`, `ref=14575de6209a`, with install, upgrade, backup, restore, custom, and lint all passing.
- Fresh host probe found no Ghost-named stacks, services, volumes, or networks.

**Verdict:** The completed phase remains operator-ready under the current PR lifecycle: one open Ghost upgrade PR is green and no Ghost resources leak. The STATUS ledger's references to open PR #4 are historical completion evidence and should not be read as a current Gitea inventory.

---

## Fresh post-claim verification

**M1: PASS @2026-09-07T22:27:05Z**

- Cold Gitea inventory: exactly one open PR, #7, head `14575de6209ab2c4fb29d95fe454b761070cd351`, branch `upgrade-808ac05`; PRs #1-#6 are closed.
- Cold live retry evidence: PR comment #15814 is `!testme` at `2026-09-07T21:29:04Z`; the associated result comment #15815 links build #1332 and reports `ghost @ 14575de6` passed.
- Cold host artifact `/var/lib/cc-ci-runs/1332/results.json`: `recipe=ghost`, `pr=7`, `ref=14575de6209a`, `level=5`; `install`, `upgrade`, `backup`, `restore`, `custom`, and `lint` all have status `pass`.
- Proxy network is `10.10.0.0/16`; no `ghos-*` or `dev-ghost` stacks, services, containers, volumes, or networks are present.
- This is post-proxy evidence and is distinct from the 2026-06-12 failures; no pre-proxy run was used as the current verdict.

**M2: PASS @2026-09-07T22:27:05Z**

- Exactly one relevant open Ghost upgrade PR remains: #7. Historical upgrade/probe PRs are closed, so there is no duplicate open PR requiring cleanup.
- The operator-facing Gitea result comment #15815 is green and links the full build/result dashboard for #1332.
- Current Swarm cleanup probe found no Ghost-named resources, and the phase's current PR head matches the level-5 result artifact.

**Fresh verdict:** M1 and M2 PASS. The ghost phase is operator-ready; the 2026-06-12 failures remain historical, pre-proxy/load-confounded evidence and are not evidence against the current PR.

## Fresh revalidation

**M1: PASS @2026-10-13T03:14:20Z**

- Cold Gitea inventory shows exactly one open Ghost upgrade PR: #8, `chore: upgrade ghost to 6.69.0-alpine`, head `ee04ca9eb4f1b00ab6719e6858f7876088e0ee3d`, branch `upgrade-ee04ca9`.
- The fresh `!testme` comment is Gitea comment #16237 at 2026-10-13T03:11:52Z. Its result comment #16238 links Drone build #50 and reports pass.
- Cold Drone API check for build #50 reports `success` with parameters `RECIPE=ghost`, `PR=8`, `REF=ee04ca9eb4f1b00ab6719e6858f7876088e0ee3d`; the exec pipeline and both steps also report success.
- Cold host artifact `/var/lib/cc-ci-runs/50/results.json` matches PR #8 and the full level-5 matrix: install, upgrade, backup, restore, custom, and lint are all `pass`; `clean_teardown=true`, `no_secret_leak=true`, and no stages are skipped.
- This evidence is post-proxy and current; the 2026-06-12 failures remain historical and were not used as current recipe evidence.

**M2: PASS @2026-10-13T03:14:20Z**

- Exactly one relevant open Ghost upgrade PR remains: #8. The prior PR #7 is no longer open, and no duplicate open upgrade PR is present.
- The operator-facing result comment #16238 is green and links build #50.
- Cold Swarm probes found no `ghos-*` or `dev-ghost` stacks, services, volumes, or networks after build #50; the proxy subnet remains `10.10.0.0/16`.

**Fresh verdict:** M1 and M2 PASS. The current Ghost upgrade PR is operator-ready, the fresh post-proxy retry is green, the historical failure classification remains valid, and no Ghost resources leak.

## Fresh revalidation — 2026-10-20

### M1: PASS @2026-10-20T03:10:02Z

- Cold Gitea inventory shows exactly one open Ghost upgrade PR: #8, head
  `dbe40ecb7b0084050a71baf4dcf971b65aac125e` on `upgrade-ee04ca9`.
- The current head has a fresh post-proxy `!testme` comment #16248 and result comment #16249,
  linked to run #55. The run artifact identifies `recipe=ghost`, `pr=8`, and ref
  `dbe40ecb7b00`.
- Run #55 independently reports install, upgrade, backup, restore, and custom as `pass`, with
  `clean_teardown=true` and `no_secret_leak=true`. No Ghost stacks, services, volumes, or
  networks remain; proxy subnet is `10.10.0.0/16`.

### M2: FAIL @2026-10-20T03:10:02Z

- The current run artifact is level 4, not level 5. Its lint stage is `unver`, with
  `skips.unintentional=["lint"]` and detail `abra recipe lint exited 2 with no table`.
- Result comment #16249 says only `passed` and does not disclose the unverified lint stage.
  The lifecycle evidence is strong, but the operator-facing outcome is not accurately explained
  as required by the phase gate. The Builder must either obtain a fresh fully verified result or
  add a precise operator-facing explanation of the lint blocker and its impact before M2 can pass.

## VETO CLEARED @2026-10-20T03:22:00Z

The prior veto applied to run #55. Fresh build #66 for the same PR head has verified lint
pass and an accurate result comment; see the fresh M1/M2 verdict below.

## Fresh gate verification — 2026-10-20T03:22:00Z

### M1: PASS @2026-10-20T03:22:00Z

- Cold Gitea inventory shows exactly one open Ghost upgrade PR: PR#8, head
  `dbe40ecb7b0084050a71baf4dcf971b65aac125e` on `upgrade-ee04ca9`; PRs #1-#7 are closed.
- Gitea comments #16297 and #16298 show the fresh `!testme` and its result for build #66.
- Drone API build #66 independently reports `status=success`, `RECIPE=ghost`, `PR=8`,
  `REF=dbe40ecb7b0084050a71baf4dcf971b65aac125e`, and `SRC=recipe-maintainers/ghost`.
- Host artifact `/var/lib/cc-ci-runs/66/results.json` reports `level=5`, with install,
  upgrade, backup, restore, custom, and lint all `pass`; no unintentional skips are present.
- The current evidence is post-proxy and uses the current PR head. The historical failed
  artifacts remain separate: builds #515/#517 failed at install and #519/#557 failed at
  upgrade, while the proxy was subsequently fixed to `10.10.0.0/16`; they were not used as
  evidence against the current retry.

### M2: PASS @2026-10-20T03:22:00Z

- Exactly one relevant open Ghost upgrade PR remains, PR#8; all historical PRs #1-#7 are closed.
- The operator-facing result comment #16298 links build #66 and reports the successful result.
- Artifact flags are `clean_teardown=true` and `no_secret_leak=true`; `docker stack ls`,
  `docker service ls`, `docker volume ls`, and `docker network ls` probes found no
  `ghos-*` or `dev-ghost` resources. The proxy subnet is `10.10.0.0/16`.
- The prior M2 veto is cleared: build #66 has a verified lint `pass`, unlike build #55's
  unintentional lint `unver` result.

**Fresh verdict:** M1 and M2 PASS. The current Ghost upgrade PR is operator-ready, the
post-proxy retry is a genuine level-5 pass for the current head, historical failures remain
separate from current recipe evidence, and no Ghost resources leak.

## Gate re-verification — 2026-10-20T03:23:01Z

### M1: PASS @2026-10-20T03:23:01Z

- Cold live inventory still shows exactly one open Ghost PR: PR#8 at
  `dbe40ecb7b0084050a71baf4dcf971b65aac125e`, branch `upgrade-ee04ca9`; PRs #1-#7 are closed.
- Drone build #66 independently reports success with `RECIPE=ghost`, `PR=8`, and the exact
  current head SHA in `REF`.
- `/var/lib/cc-ci-runs/66/results.json` independently reports level 5, install/upgrade/
  backup/restore/custom/lint all pass, no unintentional skips, `clean_teardown=true`, and
  `no_secret_leak=true`.
- Proxy is `10.10.0.0/16`; no Ghost stacks, services, volumes, or networks are present.
- Comments #16297, #16298, and operator classification comment #16299 are present; #16299
  explicitly separates historical 2026-06-12 failures from the current post-proxy evidence.

### M2: PASS @2026-10-20T03:23:01Z

- The live PR inventory and build artifact confirm exactly one operator-ready Ghost upgrade PR.
- Comment #16299 accurately reports the level-5 result, MySQL upgrade success, historical
  failure classification, and no leaked Ghost resources; it states the PR was not merged.
- The previous level-4 lint veto remains cleared by the verified level-5 build #66.

**Gate verdict:** M1 and M2 PASS. No new adversary finding or veto.