status(cfold): record fresh ghost probe

2026-06-13 00:14:11 +00:00
parent 626773d5f7
commit fb8762acb9
2 changed files with 112 additions and 12 deletions
--- a/machine-docs/JOURNAL-cfold.md
+++ b/machine-docs/JOURNAL-cfold.md
@ -248,4 +248,98 @@ Conclusion:
  unchanged, the recipe artifact still carries the expected `compose.ccci.yml` file, and the failure
  remains in the live upgrade path rather than discovery/custom-test coverage.
 - Net: cfold remains blocked on a cfold-neutral Ghost upgrade regression / flake. No repo-local code
-  change was justified by this audit alone.
+  change was justified by that audit alone.
+
+## 2026-06-13 — Ghost PR #3 fresh probe after reopen: same upgrade-only failure, plus duplicate trigger signal
+
+I looked for the smallest allowed M2 step that did not touch recipe code: reuse an existing Ghost PR head
+that had historically gone green and rerun it through the live `!testme` path.
+
+Actions taken:
+
+```bash
+$ set -a && . /srv/cc-ci/.testenv && set +a
+$ curl -fsS -u "$GITEA_USERNAME:$GITEA_PASSWORD" -X PATCH \
+    -H 'Content-Type: application/json' \
+    -d '{"state":"open"}' \
+    "https://$GITEA_URL/api/v1/repos/recipe-maintainers/ghost/pulls/3"
+# PR #3 reopened; head remains 720faa0bebc46a34857b2933df1924ccabbd4087
+
+$ curl -fsS -u "$GITEA_USERNAME:$GITEA_PASSWORD" -X POST \
+    -H 'Content-Type: application/json' \
+    -d '{"body":"!testme"}' \
+    "https://$GITEA_URL/api/v1/repos/recipe-maintainers/ghost/issues/3/comments"
+# comment 14497 created at 2026-06-13T00:07:50Z
+```
+
+Fresh live outcomes:
+
+```bash
+$ ssh cc-ci 'jq "{run_id, pr, recipe, ref, level, results, stages: (.stages | map({name,status,summary}))}" /var/lib/cc-ci-runs/568/results.json'
+{
+  "run_id": "568",
+  "pr": "3",
+  "recipe": "ghost",
+  "ref": "720faa0bebc4",
+  "level": 1,
+  "results": {
+    "backup": "pass",
+    "custom": "pass",
+    "install": "pass",
+    "restore": "pass",
+    "upgrade": "fail"
+  },
+  "stages": [
+    {"name": "install", "status": "pass", "summary": null},
+    {"name": "backup", "status": "pass", "summary": null},
+    {"name": "restore", "status": "pass", "summary": null},
+    {"name": "custom", "status": "pass", "summary": null},
+    {"name": "lint", "status": "pass", "summary": null}
+  ]
+}
+
+$ ssh cc-ci 'jq "{run_id, pr, recipe, ref, level, finished, results, stages: (.stages | map({name,status}))}" /var/lib/cc-ci-runs/569/results.json'
+{
+  "run_id": "569",
+  "pr": "3",
+  "recipe": "ghost",
+  "ref": "720faa0bebc4",
+  "level": 1,
+  "finished": 1781309502.5494862,
+  "results": {
+    "backup": "pass",
+    "custom": "pass",
+    "install": "pass",
+    "restore": "pass",
+    "upgrade": "fail"
+  },
+  "stages": [
+    {"name": "install", "status": "pass"},
+    {"name": "backup", "status": "pass"},
+    {"name": "restore", "status": "pass"},
+    {"name": "custom", "status": "pass"},
+    {"name": "lint", "status": "pass"}
+  ]
+}
+```
+
+Comment-stream evidence for duplicate triggers from one `!testme`:
+
+```bash
+$ curl -fsS -u "$GITEA_USERNAME:$GITEA_PASSWORD" \
+    "https://$GITEA_URL/api/v1/repos/recipe-maintainers/ghost/issues/3/comments?limit=20"
+# ...
+# 14497: !testme (2026-06-13T00:07:50Z)
+# 14498: cc-ci failure comment for run 568 (2026-06-13T00:08:05Z)
+# 14499: cc-ci in-progress comment for run 569 (2026-06-13T00:08:05Z)
+# 14500: cc-ci in-progress comment for run 570 (2026-06-13T00:08:05Z)
+```
+
+Takeaways:
+
+- Ghost is now freshly red post-cfold on three distinct PR heads (`720faa0b`, `d88f5801`, `d42d0f7c`), all
+  with the same upgrade-only failure shape while custom discovery stays green.
+- That further weakens any cfold-local explanation; the blocker remains in Ghost's live upgrade path.
+- There is also likely a separate trigger dedupe problem: one `!testme` comment spawned runs `568`, `569`,
+  and `570`. I did not broaden into a D1 investigation in this loop step because cfold M2 is already
+  hard-blocked by Ghost's repeated upgrade failures, but the evidence is now recorded.
--- a/machine-docs/STATUS-cfold.md
+++ b/machine-docs/STATUS-cfold.md
@ -58,8 +58,10 @@ Adversary verdict:
 Current work item:
 - full real-CI `!testme` sweep evidence is mostly assembled; one recipe (`ghost`) remains non-green for
  a cfold-neutral upgrade regression on the recipe/environment side
- fresh artifact audit confirms historical build `185` passed on the same Ghost ref now failing in
-  probe build `559`, so no cfold-local code fix has been identified in this repo yet
+- fresh follow-up probes now show the Ghost upgrade failure is not confined to PR #4 / PR #5: a reopened
+  PR #3 at ref `720faa0b` also re-failed twice post-cfold (`568`, `569`) with the same shape
+- one fresh `!testme` comment on Ghost PR #3 also spawned multiple runs (`568`, `569`, `570`), so there
+  is likely a duplicate-trigger side issue to investigate separately from cfold itself

 ### M2 baseline matrix (built from live PR heads + fresh post-cfold evidence)

@ -71,7 +73,7 @@ Current work item:
 | custom-html-tiny | PR #7 `526502ba` | 5 | 1 | build `510` -> L5 |
 | discourse | PR #2 `b7d8a244` | 5 | 3 | build `521` -> L5 |
 | drone | PR #1 `049438e1` | 5 | 1 | build `506` -> L5 |
-| ghost | PR #4 `d88f5801` | 5 | 4 | build `557` -> L1 (upgrade fail) |
+| ghost | PR #3 `720faa0b` | 5 | 4 | build `568` -> L1 (upgrade fail) |
 | hedgedoc | PR #1 `441c411c` | 5 | 2 | build `555` -> L5 |
 | immich | PR #2 `17f1649c` | 5 | 3 | build `522` -> L5 |
 | keycloak | PR #3 `bfe0d16f` | 5 | 3 | build `553` -> L5 |
@ -90,17 +92,21 @@ Current work item:

 `ghost` is the only recipe still preventing an M2 claim.

- Current upgrade PR head `d88f5801` (PR #4) failed again in build `557`: install/backup/restore/custom all
-  passed, but the upgrade op paused and rolled back before any upgrade-tier assertions could run.
- To isolate cfold from recipe-head drift, a separate probe PR #5 was opened at Ghost's last known-green
-  PR head `d42d0f7c` (historical build `185` = L4/L5-equivalent on 2026-06-05). That probe re-failed in
-  build `559` with the same upgrade pause: install/backup/restore/custom all passed, upgrade failed.
- Focused artifact audit after build `559` confirmed the same-ref comparison explicitly:
+- Current upgrade PR heads and fresh post-cfold outcomes are all red with the same stage shape:
+  - PR #3 `720faa0b`: builds `568` and `569` -> L1; install/backup/restore/custom/lint pass, upgrade fail
+  - PR #4 `d88f5801`: build `557` -> L1; install/backup/restore/custom pass, upgrade fail
+  - PR #5 `d42d0f7c`: build `559` -> L1; install/backup/restore/custom/lint pass, upgrade fail
+- Focused artifact audit still confirms the strongest same-ref comparison explicitly:
  historical build `185` (`d42d0f7c7cf9`) had `upgrade=pass`, while fresh build `559` on that same ref
  has `upgrade=fail` with the canonical `custom` stage still green.
+- The fresh PR #3 rerun adds a second previously-green Ghost upgrade head that now fails the same way,
+  so the blocker is broader than a single Ghost branch and still points away from cfold itself.
+- Side observation from the PR #3 retrigger: a single `!testme` comment at `2026-06-13T00:07:50Z` spawned
+  three new Ghost runs (`568`, `569`, `570`). `568` and `569` are already red with the same upgrade-only
+  failure; `570` was still in flight at the time of this status update.
 - Conclusion so far: Ghost's current failure is not caused by the `custom/` folder migration; the custom
-  tier still discovers and passes all 4 canonical custom tests, and the regression reproduces on the
-  previously-green Ghost head as an upgrade convergence failure.
+  tier still discovers and passes all 4 canonical custom tests, and the regression reproduces across
+  multiple Ghost PR heads as an upgrade convergence failure.

 ### Fresh Adversary state