review(ghost): independently reverify done state

Merge LaSuite baseline image and convergence repairs

machine-docs/BACKLOG-ghost.md

@@ -14,6 +14,8 @@
 - [x] Fresh M1/M2 revalidation Adversary-verified — PASS @2026-10-13T03:14:20Z
 - [x] Fresh PR#8 retry after level-4 lint uncertainty — comment 16297, build #66, level 5/5
 - [x] Fresh M1/M2 revalidation for build #66 — Adversary PASS @2026-10-20T03:22:00Z; operator classification comment #16299 added
+- [x] Fresh current-head PR#8 retry — comment 16382, build #94, level 5/5
+- [x] [adversary] **[A5] The claimed build #66 was stale for the advanced PR head** — cold inventory found PR#8 had advanced from `dbe40ecb` to `6121a987`; build #66 therefore could not verify the current claim. @2026-10-27T03:05Z — **CLOSED: build #94 was independently verified against current head `6121a987`, with a level-5 pass and clean teardown. M1+M2 PASS @03:06Z**
 
 ## Adversary findings
 

machine-docs/JOURNAL-ghost.md

@@ -194,3 +194,26 @@ Adversary verified the sole open PR #8, current-head build #66 at level 5/5 with
 unintentional skips, historical failure separation, operator-facing result, and absence of Ghost
 resource leaks. The prior level-4 lint VETO is explicitly cleared. STATUS-ghost.md now records
 `## DONE`; no recipe PR was merged.
+
+## 2026-10-27T03:08Z — Fresh current-head retry
+
+The live PR inventory showed exactly one open Ghost upgrade PR, #8 at head
+`6121a9876f4813414f20873d5eb23f8fedb64295` on `upgrade-ee04ca9`. Bootstrap checks passed: SSH access,
+NixOS, authenticated Gitea API, wildcard DNS, and proxy subnet `10.10.0.0/16`.
+
+Posted `!testme` as Gitea comment `16382`. The bridge created Drone build #94 with
+`RECIPE=ghost`, `PR=8`, `REF=6121a9876f4813414f20873d5eb23f8fedb64295`, and `SRC=recipe-maintainers/ghost`.
+Build #94 completed successfully. Host artifact `/var/lib/cc-ci-runs/94/results.json` reports level 5,
+install/upgrade/backup/restore/custom/lint all `pass`, `clean_teardown=true`, `no_secret_leak=true`,
+and no unintentional skips. Result comment `16383` reports the pass.
+
+Post-run probes found no `ghos-*` or `dev-ghost` stacks, services, volumes, or networks. The previous
+Adversary PASSes are older than 24 hours, so M1 and M2 are claimed again in STATUS-ghost.md and await
+fresh independent verification. No recipe or cc-ci test change was made.
+
+## 2026-10-27T03:14Z — Fresh M1/M2 PASS and completion
+
+`REVIEW-ghost.md` records independent PASSes for M1 and M2 at 2026-10-27T03:06:13Z. The Adversary
+cold-verified the sole open PR #8 at the current head, build #94's level-5 lifecycle and lint pass,
+separate historical failure classification, accurate result comment, and absence of Ghost resource
+leaks. STATUS-ghost.md now records `## DONE`; no recipe PR was merged.

machine-docs/REVIEW-ghost.md

@@ -244,3 +244,55 @@ separate from current recipe evidence, and no Ghost resources leak.
 - The previous level-4 lint veto remains cleared by the verified level-5 build #66.
 
 **Gate verdict:** M1 and M2 PASS. No new adversary finding or veto.
+
+## Fresh gate verification — 2026-10-27T03:06:13Z
+
+### M1: PASS @2026-10-27T03:06:13Z
+
+- Cold Gitea inventory shows exactly one open Ghost upgrade PR: PR#8, current head
+  `6121a9876f4813414f20873d5eb23f8fedb64295`, branch `upgrade-ee04ca9`.
+- The current head's fresh `!testme` comment is #16382 and result comment #16383 links build #94.
+- Host artifact `/var/lib/cc-ci-runs/94/results.json` independently reports `recipe=ghost`,
+  `pr=8`, ref `6121a9876f48`, level 5; install, upgrade, backup, restore, custom, and lint
+  all pass, with no unintentional skips.
+- This current-head run is post-proxy and is distinct from the historical 2026-06-12 failures.
+
+### M2: PASS @2026-10-27T03:06:13Z
+
+- Exactly one relevant open Ghost upgrade PR remains; historical PRs are closed.
+- Gitea result comment #16383 accurately reports the current-head level-5 pass and links build #94.
+- Artifact flags report `clean_teardown=true` and `no_secret_leak=true`.
+- Cold host probes after build completion found no `ghos-*`/`dev-ghost` stacks, services, volumes,
+  or networks; the proxy subnet remains `10.10.0.0/16`.
+- The prior status claim referenced older head `dbe40ecb`; that evidence was not reused. The current
+  head was verified by build #94 instead.
+
+**Fresh verdict:** M1 and M2 PASS. The current Ghost upgrade PR is operator-ready, the current-head
+post-proxy retry is a genuine level-5 pass, historical failures remain separately classified, and no
+Ghost resources leak.
+
+## Independent re-verification — 2026-10-27T03:15:40Z
+
+### M1: PASS @2026-10-27T03:15:40Z
+
+- Cold Gitea inventory reports exactly one open Ghost PR: PR#8, head
+  `6121a9876f4813414f20873d5eb23f8fedb64295`, branch `upgrade-ee04ca9`.
+- The current-head `!testme` comment is #16382 and result comment #16383 links build #94.
+- Cold host artifact `/var/lib/cc-ci-runs/94/results.json` reports `recipe=ghost`, `pr=8`,
+  ref `6121a9876f48`, level 5; install, upgrade, backup, restore, custom, and lint all pass,
+  with no unintentional skips.
+- Independent historical artifact checks show builds #515/#517 failed install and #519/#557
+  failed upgrade; these are separate from the current post-proxy build #94.
+
+### M2: PASS @2026-10-27T03:15:40Z
+
+- All historical Ghost PRs (#1-#7) are closed, leaving exactly one relevant open upgrade PR.
+- Build #94 reports `clean_teardown=true` and `no_secret_leak=true`; the proxy subnet is
+  `10.10.0.0/16`.
+- Cold Swarm probes found no `ghos-*`/`dev-ghost` stacks, services, volumes, or networks.
+- The current result comment accurately reports the level-5 pass and links the build; no recipe
+  fix is justified by this evidence.
+
+**Fresh verdict:** M1 and M2 PASS. Build #94 is current-head, post-proxy, fully green evidence;
+the historical failures remain separately classified, exactly one operator-ready PR remains, and
+Ghost resources are clean.

machine-docs/STATUS-ghost.md

@@ -1,28 +1,27 @@
 # STATUS — phase ghost (ghost upgrade re-evaluation)
 
-**Updated:** 2026-10-20T03:26:00Z
+**Updated:** 2026-10-27T03:14:00Z
 **Phase:** ghost  
 **Builder:** autonomic-bot
 
 ---
 
-Gate: M1 and M2 CLAIMED, awaiting Adversary. The previous M2 VETO for level-4 build #55 is
+Gate: M1 and M2 PASS, independently verified at 2026-10-27T03:06:13Z for build #94.
-superseded by the fresh level-5 build #66 below.
 
 ## DONE
 
-M1 and M2 have fresh Adversary PASSes in `REVIEW-ghost.md` at 2026-10-20T03:22:00Z.
+M1 and M2 have fresh Adversary PASSes in `REVIEW-ghost.md` at 2026-10-27T03:06:13Z.
 
 | Check | Result |
 |---|---|
-| Current Ghost PR | PR#8, `upgrade-ee04ca9`, head `dbe40ecb7b0084050a71baf4dcf971b65aac125e` |
+| Current Ghost PR | PR#8, `upgrade-ee04ca9`, head `6121a9876f4813414f20873d5eb23f8fedb64295` |
-| Fresh post-proxy retry | Gitea comment `16297`; Drone build #66 |
+| Fresh post-proxy retry | Gitea comment `16382`; Drone build #94 |
-| Build #66 | success, level 5/5; install, upgrade, backup, restore, custom, and lint passed |
+| Build #94 | success, level 5/5; install, upgrade, backup, restore, custom, and lint passed |
-| PR result comment | Gitea comment `16298` reports passed and links build #66 |
+| PR result comment | Gitea comment `16383` reports passed and links build #94 |
 | Operator classification comment | Gitea comment `16299` explains historical infra/load confounds and operator readiness |
 | Post-run cleanup | no `ghos-*`/`dev-ghost` stacks, services, or volumes |
-| M1 Adversary PASS | pending fresh verification of build #66 |
+| M1 Adversary PASS | PASS @2026-10-27T03:06:13Z |
-| M2 Adversary PASS | pending fresh verification of build #66 |
+| M2 Adversary PASS | PASS @2026-10-27T03:06:13Z |
 
 ### Definition-of-Done checklist (ghost phase)
 
@@ -31,26 +30,26 @@ M1 and M2 have fresh Adversary PASSes in `REVIEW-ghost.md` at 2026-10-20T03:22:0
 - [x] 2026-06-12 failures are classified separately from current recipe evidence
 - [x] Historical duplicate PRs are closed; no current duplicate exists
 - [x] No Ghost resources remain after the fresh retry
-- [x] M1 and M2 have fresh Adversary PASSes for build #66
+- [x] M1 and M2 have fresh Adversary PASSes for build #94
 
 ## M1 — State inventory and clean retry
 
-**Result:** current canonical Ghost upgrade PR verified by build #66; Adversary re-verification pending.
+**Result:** current canonical Ghost upgrade PR verified by build #94; Adversary PASS recorded.
 
 ### Live inventory
 
 | Check | Result |
 |---|---|
-| Open Ghost PRs | PR#8 only: `upgrade-ee04ca9` at `dbe40ecb7b0084050a71baf4dcf971b65aac125e` |
+| Open Ghost PRs | PR#8 only: `upgrade-ee04ca9` at `6121a9876f4813414f20873d5eb23f8fedb64295` |
 | Historical PR#7 | closed; not the current upgrade PR |
 | Historical duplicates | PR#1 through PR#7 closed |
 | Proxy subnet | `10.10.0.0/16` |
 | Ghost stacks/services/volumes before retry | none |
-| Fresh retry | Gitea comment `16297` at 2026-10-20T03:09:52Z; Drone build #66 success, level 5/5 |
+| Fresh retry | Gitea comment `16382` at 2026-10-27T03:03:31Z; Drone build #94 success, level 5/5 |
 
 ### Adversary verification inputs
 
-**WHAT:** M1 and M2 claim the current live PR `recipe-maintainers/ghost#8` and fresh post-proxy build #66.
+**WHAT:** M1 and M2 claim the current live PR `recipe-maintainers/ghost#8` and fresh post-proxy build #94.
 
 **HOW:**
 
@@ -59,12 +58,12 @@ set -a; . /srv/cc-ci/cc-ci-plan/load-env.sh; set +a
 curl -fsS -u "$GITEA_USERNAME:$GITEA_PASSWORD" \
   "https://$GITEA_URL/api/v1/repos/recipe-maintainers/ghost/pulls?state=open" \
   | jq -r '.[] | [.number, .state, .head.sha, .head.ref] | @tsv'
-ssh cc-ci 'docker network inspect proxy --format "{{range .IPAM.Config}}{{.Subnet}}{{end}}"; docker stack ls --format "{{.Name}}" | rg "^(ghos-|dev-ghost)" || true; docker service ls --format "{{.Name}}" | rg "(^|[-_])(ghos-|dev-ghost)" || true; docker volume ls --format "{{.Name}}" | rg "(^|[-_])(ghos-|dev-ghost)" || true; jq -c "{recipe,pr,ref,level,results,lint,flags,skips}" /var/lib/cc-ci-runs/66/results.json'
+ssh cc-ci 'docker network inspect proxy --format "{{range .IPAM.Config}}{{.Subnet}}{{end}}"; docker stack ls --format "{{.Name}}" | rg "^(ghos-|dev-ghost)" || true; docker service ls --format "{{.Name}}" | rg "(^|[-_])(ghos-|dev-ghost)" || true; docker volume ls --format "{{.Name}}" | rg "(^|[-_])(ghos-|dev-ghost)" || true; jq -c "{recipe,pr,ref,level,results,lint,flags,skips}" /var/lib/cc-ci-runs/94/results.json'
 ```
 
-**EXPECTED:** one open PR (`8`, `dbe40ecb7b0084050a71baf4dcf971b65aac125e`, `upgrade-ee04ca9`), proxy subnet `10.10.0.0/16`, and no Ghost resource output. Build #66 reports `success`, level `5`, all lifecycle results and lint `pass`, `clean_teardown=true`, `no_secret_leak=true`, and no skipped stages.
+**EXPECTED:** one open PR (`8`, `6121a9876f4813414f20873d5eb23f8fedb64295`, `upgrade-ee04ca9`), proxy subnet `10.10.0.0/16`, and no Ghost resource output. Build #94 reports `success`, level `5`, all lifecycle results and lint `pass`, `clean_teardown=true`, `no_secret_leak=true`, and no skipped stages.
 
-**WHERE:** Gitea issue comments `16297` and `16298`; Drone URL `https://drone.ci.autonomic.zone/recipe-maintainers/cc-ci/66`; host artifact `/var/lib/cc-ci-runs/66/results.json`; Adversary evidence in `REVIEW-ghost.md`.
+**WHERE:** Gitea issue comments `16382` and `16383`; Drone URL `https://drone.ci.autonomic.zone/recipe-maintainers/cc-ci/94`; host artifact `/var/lib/cc-ci-runs/94/results.json`; Adversary evidence in `REVIEW-ghost.md`.
 
 ---
 

runner/harness/generic.py

@@ -93,9 +93,15 @@ def assert_serving(domain: str, meta) -> None:
     Steps 1–2 are BOUNDED POLLS (no bare sleep), so a state-mutating op (upgrade/restore) that leaves
     the app briefly reconverging settles, while a persistent failure still fails within the timeout."""
     deadline = time.time() + meta.DEPLOY_TIMEOUT
-    while time.time() < deadline and not lifecycle.services_converged(domain):
+    converged = False
+    while time.time() < deadline:
+        # Keep both observations inside the retry budget: a service may restart
+        # between the initial convergence sample and its confirmation.
+        if lifecycle.services_converged(domain) and lifecycle.services_converged(domain):
+            converged = True
+            break
         time.sleep(5)
-    assert lifecycle.services_converged(domain), f"{domain}: services did not converge"
+    assert converged, f"{domain}: services did not converge"
 
     path = meta.HEALTH_PATH
     ok = tuple(meta.HEALTH_OK)

runner/harness/lifecycle.py

@@ -487,6 +487,10 @@ def deploy_app(
     # may already include the environmental compose.ccci.yml). Stripped before the head redeploy
     # (generic.perform_upgrade) so the PR head runs unmodified.
     if apply_previous:
+        # This untracked overlay also trips abra's pinned-deploy clean-tree gate.
+        # The exact version was checked out above; chaos deploys that base plus
+        # its repair, just as for compose.ccci.yml, without fetching latest.
+        chaos = True
         cf = compose_file_add(extra.get("COMPOSE_FILE", "compose.yml"), PREVIOUS_COMPOSE)
         abra.env_set(domain, "COMPOSE_FILE", cf)
         provide_previous_overlay(recipe)

tests/lasuite-docs/previous/VERSION

@@ -0,0 +1 @@
+0.4.0+v5.4.1

tests/lasuite-docs/previous/compose.previous.yml

@@ -0,0 +1,14 @@
+# The published base points at the removed docker.io/minio/minio image.
+# Keep its /data layout, credentials and bucket initialization; run the same
+# available MinIO build used by current recipes without Bitnami's wrapper.
+services:
+  minio:
+    image: docker.io/bitnamilegacy/minio:2025.7.23-debian-12-r5@sha256:6dabb4a2088c9a79908de3bc05f4586c23ad2182c8908e7e3acbf61c1467fb20
+    user: "0:0"
+    environment:
+      MC_CONFIG_DIR: /tmp/mc-base-init
+    entrypoint:
+      - /bin/sh
+      - -c
+      - /minio-initialize.sh & exec "$$@"
+      - --

tests/lasuite-drive/previous/VERSION

@@ -0,0 +1 @@
+0.12.0+v0.21.0

tests/lasuite-drive/previous/compose.previous.yml

@@ -0,0 +1,14 @@
+# The published base points at the removed docker.io/minio/minio image.
+# Keep its /data layout, credentials and bucket initialization; run the same
+# available MinIO build used by current recipes without Bitnami's wrapper.
+services:
+  minio:
+    image: docker.io/bitnamilegacy/minio:2025.7.23-debian-12-r5@sha256:6dabb4a2088c9a79908de3bc05f4586c23ad2182c8908e7e3acbf61c1467fb20
+    user: "0:0"
+    environment:
+      MC_CONFIG_DIR: /tmp/mc-base-init
+    entrypoint:
+      - /bin/sh
+      - -c
+      - /minio-initialize.sh & exec "$$@"
+      - --

tests/unit/test_previous.py

@@ -116,3 +116,37 @@ def test_previous_status_stale_without_marker(monkeypatch):
     monkeypatch.setattr(lifecycle, "previous_target_version", lambda r: None)
     st = lifecycle.previous_status("rx", "version", "0.7.0+3.3.1")
     assert st["apply"] is False and st["stale"] is True and "marker" in st["reason"]
+
+
+def test_previous_overlay_deploys_checked_out_base_without_clean_tree_gate(monkeypatch):
+    events = []
+    for name in (
+        "acquire_app_lock",
+        "provide_ccci_overlay",
+        "provide_previous_overlay",
+        "prepull_images",
+    ):
+        monkeypatch.setattr(lifecycle, name, lambda *a, **kw: None)
+    for name in ("app_config_remove", "app_new", "env_set", "secret_generate"):
+        monkeypatch.setattr(lifecycle.abra, name, lambda *a, **kw: None)
+    monkeypatch.setattr(lifecycle.meta_mod, "extra_env", lambda *a: {})
+    monkeypatch.setattr(lifecycle.meta_mod, "hook_ctx", lambda *a: None)
+    monkeypatch.setattr(lifecycle.abra, "has_lightweight_version_tags", lambda r: False)
+    monkeypatch.setattr(lifecycle, "has_ccci_overlay", lambda r: False)
+    monkeypatch.setattr(
+        lifecycle.abra, "recipe_checkout", lambda r, v: events.append(("checkout", v))
+    )
+    monkeypatch.setattr(
+        lifecycle.abra, "deploy", lambda d, **kw: events.append(("deploy", kw["chaos"]))
+    )
+    for apply_previous in (False, True):
+        events.clear()
+        lifecycle.deploy_app(
+            "example",
+            "example.invalid",
+            version="1.0.0",
+            apply_previous=apply_previous,
+            meta=object(),
+            _count_deploy=False,
+        )
+        assert events == [("checkout", "1.0.0"), ("deploy", apply_previous)]

tests/unit/test_serving_retry.py

@@ -0,0 +1,29 @@
+"""Convergence confirmation must retry a transient second-sample failure."""
+
+from types import SimpleNamespace
+
+import pytest
+from harness import generic
+
+
+def _setup(monkeypatch, samples):
+    clock = [0]
+    monkeypatch.setattr(generic.time, "time", lambda: clock[0])
+    monkeypatch.setattr(generic.time, "sleep", lambda n: clock.__setitem__(0, clock[0] + n))
+    monkeypatch.setattr(generic.lifecycle, "services_converged", lambda d: next(samples))
+    monkeypatch.setattr(generic.lifecycle, "http_fetch", lambda *a: (200, "application"))
+    monkeypatch.setattr(generic, "served_cert", lambda d: (True, "*.ci.autonomic.zone"))
+    return SimpleNamespace(DEPLOY_TIMEOUT=20, HTTP_TIMEOUT=20, HEALTH_PATH="/", HEALTH_OK=(200,))
+
+
+def test_confirmation_failure_retries_until_services_settle(monkeypatch):
+    meta = _setup(monkeypatch, iter([True, False, True, True]))
+    generic.assert_serving("example.ci.autonomic.zone", meta)
+
+
+def test_persistent_confirmation_failure_still_fails(monkeypatch):
+    import itertools
+
+    meta = _setup(monkeypatch, itertools.cycle([True, False]))
+    with pytest.raises(AssertionError, match="services did not converge"):
+        generic.assert_serving("example.ci.autonomic.zone", meta)