cc-ci/machine-docs/REVIEW-ghost.md

REVIEW — phase ghost (Adversary)

Cold reconnaissance — 2026-06-13T06:20Z

Scope: Pre-Builder independent probe of ghost PR/build state.
Source of truth: phase plan plan-phase-ghost-reeval.md §Gates / DoD.

What was checked

• Gitea API: all open/closed PRs on recipe-maintainers/ghost
• ci.commoninternet.net ghost run history: builds #515–#585
• Drone build logs (read directly via Drone sqlite DB): builds #557, #578, #585
• cc-ci host: docker stacks/volumes/services matching "ghost"
• /tmp/ghost-render/compose.ccci.yml overlay contents

Pre-claim findings

F1 — Upgrade failure mode is MySQL timing, NOT VIP exhaustion.
Builds #557 and #578 both show: "!! upgrade op failed: ... UpdateStatus='paused'" — recipe-level timing failure. Not VIP exhaustion (which would be tasks stuck in New state).

F2 — Build #585 pre-proxy, wrong PR. Ran at ~04:14Z (84 min before proxy fix at 05:38Z). Tested PR#5 (d42d0f7c), not PR#4 (d88f5801).

F3 — No post-proxy ghost runs as of 06:20Z. Builder needed to trigger a fresh run.

F4 — MySQL timing is load-sensitive. Same sha: #578 failed at ~03:00Z, #585 passed at ~04:00Z. Suggests server load was the variable.

F5 — PR#5 is cfold artifact. Should be closed after PR#4 verdict.

F6/F7 — Clean state. No ghost leaks; all recent runs have clean_teardown=true, no_secret_leak=true.

---

M1 — State inventory and clean retry

PASS @2026-06-13T06:38Z

Cold acceptance run

Adversary independently verified the following from a cold start (own clone, own SSH session, no Builder state shared):

1. Correct PR identified: PR#4 (d88f5801)

• Gitea API confirms PR#4 is the only open PR, titled "chore: upgrade to 1.4.0+6.44.1-alpine"
• PR#5 (cfold probe) now closed ✅

2. Pre-proxy failures confirmed infra-confounded

• Builds 515, 517, 519, 557: all dated 2026-06-12, before proxy /16 fix at 05:38Z on 2026-06-13 ✅
• Builds 515/517 were L0 (possible VIP exhaustion at deploy stage); builds 519/557 were L1 with UpdateStatus=paused (MySQL timing under high load from concurrent IPAM-fix operations)
• Builder's classification as "infra-confounded" is correct

3. Fresh post-proxy !testme on PR#4 verified

• Gitea PR#4 comment: @autonomic-bot [2026-06-13T06:12:48Z]: !testme (post-proxy ✅, proxy fixed 05:38Z)
• Drone build #612: started=2026-06-13T06:13:02Z (from Drone sqlite DB) — 35 min after proxy fix ✅
• RECIPE=ghost REF=d88f5801 ✅
• build_status=success ✅

4. Build #612 genuine L5/5 pass verified

• /var/lib/cc-ci-runs/612/results.json: level=5, all stages pass (install/upgrade/backup/restore/custom) ✅
• JUnit timestamps confirm genuine sequential execution:
  • install: 06:13:53Z (51s from start)
  • upgrade: 06:14:38Z (1m36s from start)
  • backup: 06:14:43Z
  • restore: 06:14:49Z
  • custom: 06:14:50–53Z
• clean_teardown=True, no_secret_leak=True ✅
• Badge: https://ci.commoninternet.net/runs/612/badge.svg → level 5 ✅
• Proxy subnet confirmed: 10.10.0.0/16 ✅

Evidence source: all checks run independently by Adversary against Gitea API, cc-ci Drone sqlite, cc-ci run log files, and cc-ci docker state.

---

M2 — Operator-ready outcome

PASS @2026-06-13T06:38Z

Cold acceptance run

1. Exactly 1 open PR on ghost: PR#4

• GET /api/v1/repos/recipe-maintainers/ghost/pulls?state=open → 1 result: PR#4 (d88f5801) ✅

2. PR#3 closed

• GET /api/v1/repos/recipe-maintainers/ghost/pulls/3 → state=closed ✅

3. PR#5 closed

• GET /api/v1/repos/recipe-maintainers/ghost/pulls/5 → state=closed ✅

4. No ghost resource leaks

• docker stack ls | grep ghos = nothing ✅
• docker service ls | grep ghos = nothing ✅
• docker volume ls | grep ghos = nothing ✅

5. Operator comment on PR#4

• Comment at 2026-06-13T06:22:11Z (note: STATUS says 06:35Z — minor discrepancy, not blocking)
• Content: 5-tier pass table, infra-confound analysis, "This PR is operator-ready. Nothing was merged." ✅

6. Adversary findings from BACKLOG addressed:

• A1: Build #585 NOT used as post-proxy pass — Builder used #612 (post-proxy) ✅
• A2: MySQL timing acknowledged in operator comment; upgrade passed post-proxy confirming infra-confound ✅
• A3: PR#5 closed ✅

Verdict

Both M1 and M2 PASS. The ghost phase Definition of Done is met:

• Exactly one ghost upgrade PR (PR#4) is operator-ready
• Fresh post-proxy verdict: PASS (build #612, level 5/5)
• 2026-06-12 failures correctly classified as infra-confounded (proxy /24 IPAM pressure + load)
• No stale stacks/volumes
• Operator-facing explanation present on the PR

Builder may write ## DONE to STATUS-ghost.md.