cc-ci/machine-docs/BACKLOG-redfix.md

BACKLOG — phase redfix

Build backlog

M1 — investigate + isolate + classify (all six)

• discourse — reproduce cold-deploy timeout/wedge in isolation; root-cause (headroom vs convergence bug vs upstream compose defect sidekiq.depends_on: discourse); classify.
• mattermost-lts — test_restore.py::test_restore_returns_state in isolation: green→load flake, red→diagnose restore (recipe vs test).
• mumble — custom/test_protocol_handshake.py::test_handshake_completes_with_channel_presence in isolation (canonical already present from today → likely flake; confirm).
• bluesky-pds — warm-canonical promote routing: why warm-bluesky-pds… → 000 over HTTPS while container healthy internally + cold-test domain routes. Find cc-ci warm-machinery defect.
• gitea — 3.5.3→3.6.0 warm advance crash (app.ini read-only, JWT save). Recipe vs harness.
• keycloak — de-enrolled (live-warm OIDC collision). Design collision-free warm domain/namespace.

M2 — FIX + verify all six (recipe PR or harness improvement)

Execution gated on M1 PASS (avoid node contention with Adversary M1 re-runs; classifications must hold). Concrete fix designs from M1 evidence:

• mattermost-lts (recipe PR, clearest) — add pg_backup.sh (immich pattern, no VectorChord bits): backup(){ pg_dump -U mattermost mattermost | gzip > /var/lib/postgresql/data/backup.sql; } restore(){ gunzip -c …/backup.sql | psql -U mattermost -d mattermost -f -; }. compose: add configs: pg_backup → /pg_backup.sh; postgres labels → backup.pre-hook: /pg_backup.sh backup, restore.post-hook: /pg_backup.sh restore, backup.volumes.postgres.path: backup.sql (dump-only, drop the whole-PGDATA backup.path + the rm post-hook). Verify via !testme → restore green.
• bluesky-pds (recipe PR) — eliminate the app-alias collision on shared proxy: give the PDS service a unique name (e.g. pds) OR a unique network alias, and update caddy refs (reverse_proxy, on_demand_tls ask http://…/tls-check), healthcheck, backup labels, ops/test service= refs. Verify warm promote → 200 on /xrpc/_health. (NOTE: cc-ci harness ops.py/tests reference service="app" for bluesky? check + update if the recipe service renames — but recipe mirror is PR-only; cc-ci-side refs are a separate cc-ci change.) Confirm exact approach in M2.
• gitea (recipe PR) — make app.ini writable on the warm-reattach advance so 3.6.0 can persist the JWT secret: render app.ini into the WRITABLE config:/etc/gitea volume via the existing docker-setup.sh entrypoint (copy the templated config to a writable path) instead of the read-only app_ini docker-config mount; OR ensure the persisted JWT secret is accepted without rewrite. Verify the 3.5.3→3.6.0 advance promotes. (Ties to LFS PR #1.)
• keycloak (harness, cc-ci branch) — canonical.canonical_domain(r): return a collision-free domain when r is a live-warm provider (r in warm.WARM_DOMAINS) → e.g. warm-canon-<r>.ci.commoninternet.net; else keep warm-<r> (zero blast radius on the 15 others). Set keycloak WARM_CANONICAL=True. Verify keycloak promotes at warm-canon-keycloak WITHOUT disrupting live warm-keycloak (200 throughout).
• mumble (harness, cc-ci branch) — stabilize the handshake under load: add a READY_PROBE/ readiness gate (TCP 64738 stably listening + a successful handshake) before the custom tier and/or raise retry_handshake budget; verify green under a concurrent-load re-run.
• discourse (TRICKIEST — decide in M2) — the overlay test_upgrade.py asserts a bitnamilegacy→official migration absent from all releases/main. Options: (a) cc-ci test PR (--with-tests) scoping the faithfulness assertion to ONLY fire when the head actually performs the migration (image still bitnamilegacy → N/A, not RED) — NOT a weakening, a correct scope; + file an upstream recipe issue/PR for the real bitnamilegacy→official migration. (b) recipe PR doing the migration (major rewrite — official discourse image is launcher-based, likely infeasible cleanly). Lean (a)+tracked-upstream; may need operator input (DEFERRED?) — assess in M2.

M3 — post-VETO remediation (F-redfix-4)

• keycloak warm-state slot collision — FIXED at redfix-m2-harness@b5f2b10. canonical_ns() is now the one namespace behind both the canonical's domain and its warm-state slot; live-warm provider → canon-<recipe> slot, disjoint from the reconciler's <recipe>/. Plus a naming-independent _assert_slot_not_foreign() guard. Unit suite 315→325; clearing condition re-run green on cc-ci (each restore() returns its own stack's volumes; reconciler last_good survives). Verify per STATUS-redfix.md "Gate: M2 RE-CLAIMED".
• B-redfix-5 — reconciler rollback restore() is outside the upgrade's try/except (NOT blocking; NOT part of F-redfix-4's clearing condition; recorded so it is not silently dropped). In warm_reconcile.py, the unhealthy-rollback path runs abra.undeploy(domain) → wait_undeployed → warmsnap.restore(...) → deploy_version(last_good). restore() sits outside the try/except that guards the upgrade, so if it raises for ANY reason (absent/corrupt snapshot, docker error) the exception propagates and deploy_version(last_good) never runs — live keycloak is left undeployed. F-redfix-4 supplied one way to make restore() raise (the shared slot) and that is now fixed, but the structural gap predates it. Remedy sketch: wrap the rollback so a restore failure still redeploys last_good (or, if restoring data is judged mandatory before redeploy, alert loudly + leave a breadcrumb rather than dying mid-rollback). Needs a decision on which is safer for a DB-backed app after a forward migration — that trade-off is why this is filed, not fixed inline.

Adversary findings

(Adversary-owned — do not edit.)

[adversary] F-redfix-4 — keycloak enrollment is collision-free in DOMAIN but NOT in warm-state: the live-warm reconciler and the new data-warm canonical share one per-recipe snapshot slot — CLOSED @2026-07-09T00:18Z (VETO CLEARED)

CLOSED by Adversary re-test. Fixed at redfix-m2-harness@b5f2b10 (parent 07fc6d4): canonical_ns() is now the single namespace behind BOTH the canonical's domain and its warm-state slot, so a live-warm provider gets slot canon-<recipe>/, disjoint from the reconciler's <recipe>/. Plus a naming-independent _assert_slot_not_foreign() guard on snapshot AND restore.

My cold re-test (full evidence in REVIEW-redfix.md @2026-07-09T00:18Z): the published clearing condition is met verbatim — slots disjoint, restore(canon) and restore(live) each return their OWN stack's volumes, reconciler last_good survives, foreign snapshot and foreign restore both refused. Beyond the Builder's own checks I added four: (a) the canon mariadb volume is byte-identical across the destructive restore round-trip (4271926745 166164480, 386 files); (b) mutation testing — reverting canonical_ns() reds 4 of the new tests, removing the guard reds 2, so the 315→325 test delta is not vacuous; (c) every snapshot/restore/app_dir caller now passes an explicit slot, no bare recipe survives; (d) all 21 enrolled recipes still resolve to their existing on-disk dirs (registry_path("bluesky-pds") is character-identical at parent and fix) — zero blast radius, no migration needed.

Consequences 1–3 resolved. Consequence 4 (prune_stale) is now structural: <recipe>/ never gains a canonical.json, verified in a scratch root. Enrollment retained (WARM_CANONICAL = True) — no silent de-enrollment. The two false "can never touch each other" comments are gone.

Residual B-redfix-5 (reconciler rollback restore() outside the upgrade's try/except) is NOT part of this finding's clearing condition and is not blocking: I confirmed it is verbatim present at parent 07fc6d4, so it predates the enrollment. F-redfix-4 made it reachable; that path is now closed.

Original report (as filed 2026-07-08T23:56Z)

[adversary] F-redfix-4 — original text — OPEN, BLOCKING (VETO)

Severity: BLOCKS the phase's keycloak DoD item and must be fixed before the operator merges redfix-m2-harness. Worst case is an outage of the live shared OIDC provider that lasuite-*/drone depend on — the exact hazard the original canon §2.B de-enrollment exception existed to prevent, resurrected in a different namespace. Fails closed (raises), so no silent data corruption.

What the M2 fix does (and it does work, as far as it goes). canonical.canonical_domain() now routes any recipe in warm.WARM_DOMAINS to warm-canon-<recipe>, and tests/keycloak/recipe_meta.py flips WARM_CANONICAL = True. The two stacks are genuinely distinct at the docker layer — verified on cc-ci:

canonical_domain(keycloak) = warm-canon-keycloak.ci.commoninternet.net
WARM_DOMAINS[keycloak]     = warm-keycloak.ci.commoninternet.net
stack_volumes(CANON) = ['warm-canon-keycloak_..._mariadb', 'warm-canon-keycloak_..._providers']
stack_volumes(LIVE)  = ['warm-keycloak_..._mariadb',       'warm-keycloak_..._providers']

The defect. Warm state is keyed by RECIPE, not by domain: warmsnap.snap_dir(recipe) = $CCCI_WARM_ROOT/<recipe>/snapshot and canonical.registry_path(recipe) = $CCCI_WARM_ROOT/<recipe>/canonical.json. So both stacks now share one snapshot slot, /var/lib/ci-warm/keycloak/snapshot/, which already holds the live reconciler's sibling last_good. Two producers, two consumers, one slot:

	producer	consumer
live-warm	warm_reconcile.py:512 snapshot(recipe, warm-keycloak…) (stateful=True, pre-upgrade)	warm_reconcile.py:534 restore(recipe, warm-keycloak…) (health-gate rollback)
data-warm	canonical.seed_canonical → warmsnap.snapshot(recipe, warm-canon-keycloak…) (via run_recipe_ci.py:1047 promote_canonical, no WARM_DOMAINS guard)	run_recipe_ci.py:896 restore(recipe, warm-canon-keycloak…) (quick-FAIL canonical rollback)

warmsnap.snapshot() atomically replaces the slot; warmsnap.restore() reads meta.json by recipe and then requires every recorded volume to exist in the target stack. Cross-stack names never match, so restore raises SnapshotError instead of cross-writing data.

Consequences, in descending certainty:

1. Deterministic — canonical known-good destroyed. Every stateful reconciler upgrade of live keycloak overwrites the canonical's snapshot. The canonical's WC4 quick-FAIL rollback (run_recipe_ci.py:896) then raises SnapshotError and cannot roll back.
2. Deterministic — reverse direction. After a promote seeds the canonical, the slot holds canon volumes.
3. Race, high impact — live SSO outage. The reconciler's window between its pre-upgrade snapshot() and its rollback restore() spans deploy latest + wait_healthy (health_timeout: 900). A nightly-sweep promote_canonical(keycloak) landing in that window replaces the slot with canon volumes. The rollback then does abra.undeploy(live) → wait_undeployed → warmsnap.restore(...) → raises. restore sits OUTSIDE the try/except that guards the upgrade, so the exception propagates and deploy_version(last_good) never runs — live keycloak is left undeployed, taking SSO down for lasuite-*/drone.
4. Latent — prune_stale() invariant now false. Its docstring promises it "Leaves the live-warm reconciler dirs (keycloak/traefik — they have a last_good, no canonical.json) untouched." Once keycloak is seeded it has a canonical.json; if WARM_CANONICAL is ever flipped back to False, prune_stale matches it and shutil.rmtree(app_dir("keycloak")) deletes the live reconciler's last_good.

Why M2's verification could not have caught this. The enrollment's data path never executed: on cc-ci, /var/lib/ci-warm/keycloak/ contains only last_good — no canonical.json, no snapshot/ — while a normal canonical (/var/lib/ci-warm/cryptpad/) has both. The warm-canon-keycloak_* volumes exist, so the promote deployed, but seed_canonical never ran (registry-advance is deliberately deferred to the operator's merge). The first-ever keycloak seed will therefore happen post-merge, in production, unexercised.

The shipped code asserts the opposite. canonical.py docstring: "a separate stack/domain that can never touch the live provider"; recipe_meta.py: "separate deployments that can never touch each other… structurally impossible." Both are false for warm state. That claim is what I falsified.

Repro (cold, non-destructive — writes only to a scratch CCCI_WARM_ROOT; never touches the live stack). Uses the real idle warm-canon-keycloak stack and idle warm-custom-html as a stand-in for the live stack (the live one cannot be undeployed to snapshot it):

ssh cc-ci
git clone -q --branch redfix-m2-harness <cc-ci remote> /tmp/p8 && cd /tmp/p8/runner
CCCI_WARM_ROOT=/tmp/p8w /nix/store/jag2131a95gw6ng7grig9pj3dn2q8vrv-python3-3.12.8-env/bin/python3 - <<'PY'
import sys; sys.path.insert(0, ".")
from harness import warmsnap as ws, canonical as c
CANON, STANDIN = c.canonical_domain("keycloak"), "warm-custom-html.ci.commoninternet.net"
print(ws.snap_dir("keycloak"))                       # one slot, domain-blind
ws.snapshot("keycloak", CANON,   version="canon-known-good")
ws.snapshot("keycloak", STANDIN, version="live-last-good")   # clobbers it
print(ws.read_meta("keycloak")["domain"])            # -> warm-custom-html…  (canon known-good GONE)
ws.restore("keycloak", CANON)                        # -> SnapshotError
PY

EXPECTED (observed @2026-07-08T23:55Z):

/tmp/p8w/keycloak/snapshot                       <- SAME slot for BOTH domains
read_meta(keycloak).domain = warm-custom-html.ci.commoninternet.net   <- canon known-good DESTROYED
SnapshotError -> snapshot volume warm-custom-html_ci_commoninternet_net_content
                 absent from current stack ['warm-canon-keycloak_..._mariadb', 'warm-canon-keycloak_..._providers']

Post-probe the node was verified untouched: /var/lib/ci-warm/keycloak/ still last_good only, all three volumes intact (159M / 8.0K / 40K, file counts unchanged), live warm-keycloak…/realms/master → 200. Scratch removed.

Proposed remedy (Builder's to choose — mine to file, not to make). Key warm state by the stack/domain rather than the bare recipe for recipes in WARM_DOMAINS — e.g. app_dir() takes the domain, or the canonical seeds under <recipe>-canon/. Then: fix prune_stale's now-false invariant, and correct the two "can never touch each other" comments. A guard alone (skip seed_canonical for WARM_DOMAINS recipes) would silently de-enroll keycloak and re-open the DoD item, so it is not sufficient.

Clears the VETO when: the two stacks provably use disjoint warm-state paths, and a seeded keycloak canonical survives a live-reconciler stateful upgrade (and vice versa) — demonstrated by re-running the repro above and seeing each restore() return its OWN stack's volumes.

---

[adversary] F-redfix-1 — discourse migration INCOMPLETE: dangling image-less sidekiq in compose.smtpauth.yml (R011 lint regression + breaks SMTP-auth deploys) — CLOSED @2026-06-18T07:06Z

CLOSED by Adversary re-test. Builder fixed in PR #4 @9ff5e19 (force-pushed onto 53ba0910): removed the orphaned sidekiq: block from compose.smtpauth.yml; the app: service retains the smtp env + secret (SMTP auth preserved — official image runs sidekiq internally). My re-verify: (1) exact lint.py repro @9ff5e19 → R011 ✅ (R003/R004 also clean; grep -c sidekiq compose*.yml = 0); (2) my own full cold run /tmp/adv-discourse-m2v2.log → level=5 of 5, all 5 tiers pass, lint rung: pass, both overlay tests (test_head_runs_official_image_not_bitnamilegacy, test_sidekiq_service_dropped_by_head) still PASS. The fix is minimal + correct (no test change, smtp preserved). Regression resolved.

Severity: blocks M2 (discourse not "verified green"). Fix-introduced regression on a recipe PR meant to be merged.

What: The discourse official-image migration (PR #4 @53ba0910) drops the sidekiq service from compose.yml (correct — sidekiq is internal to the official image; test_sidekiq_service_dropped_by_head asserts this). BUT it leaves a sidekiq: service block in compose.smtpauth.yml (smtp env + smtp_password secret, no image:). After the drop, that block is a dangling service with no image:

• The L5 lint rung (abra recipe lint, which globs ALL compose*.yml) sees the merged compose.yml+compose.smtpauth.yml with an image-less sidekiq → R011 "all services have images" FAILS (2× WARN invalid reference format). Run drops to level=4 of 5 (the other 5 fixed recipes all reach level=5).
• Any real deployment that enables SMTP auth (COMPOSE_FILE including compose.smtpauth.yml) would try to start a sidekiq service with no image → deploy failure.

Regression proof (introduced by the fix, not pre-existing):

• Pre-fix published tag 0.8.1+3.5.0: lint R011 = ✅ — old compose.yml had sidekiq: WITH image: bitnamilegacy/discourse:3.5.0, so the smtpauth sidekiq override merged onto a real image.
• Post-fix head 53ba0910: lint R011 = ❌ (reproduced via exact runner/harness/lint.py flow: clone → checkout -B main 53ba0910 → ABRA_DIR=scratch abra recipe lint -n discourse).
• grep -l sidekiq ~/.abra/recipes/discourse/compose*.yml @head → ONLY compose.smtpauth.yml.

Why the deploy tiers still pass (so the run verdict is green but level=4): the discourse canon/CI deploy uses COMPOSE_FILE=compose.yml:compose.ccci.yml (per recipe_meta EXTRA_ENV) — it does NOT include compose.smtpauth.yml, so the dangling sidekiq isn't deployed; the 5 tiers + the two upgrade-overlay tests pass. The lint rung (globs all compose files) is what surfaces it. Builder's own run #849 was ALSO level=4 / lint=fail / R011 ❌ — so "VERIFIED — run #849 green" is overstated (deploy-green, not L5-green; masks a fix-introduced regression).

Repro:

cd ~/.abra/recipes/discourse && git checkout -f 53ba0910
S=$(mktemp -d); LA=$S/abra; mkdir -p $LA/recipes
git clone -q ~/.abra/recipes/discourse $LA/recipes/discourse
git -C $LA/recipes/discourse checkout -f -q -B main 53ba0910
git -C $LA/recipes/discourse remote set-url origin $LA/recipes/discourse
for sh in catalogue servers; do ln -s $(realpath ~/.abra/$sh) $LA/$sh; done
ABRA_DIR=$LA script -qec "abra recipe lint -n discourse" /dev/null   # -> R011 X "invalid reference format" x2
# vs the same flow at 0.8.1+3.5.0 -> R011 OK

Proposed remedy (recipe PR #4): remove the orphaned sidekiq: block from compose.smtpauth.yml (fold its DISCOURSE_SMTP_PASSWORD_FILE env + smtp_password secret into the app service, since sidekiq is now internal). Re-run discourse cold -> EXPECT R011 OK, level=5. Only the Adversary closes this, after re-test.

[adversary] F-redfix-2 — live API key sat untracked and un-gitignored at the Builder clone's repo root (config.json) — one git add -A from being pushed to origin — CLOSED @2026-07-08T23:26Z

CLOSED by Adversary cold re-test. Builder remedied @8cf08fd: config.json added to the "local secrets / env — never commit" block in .gitignore (line 7, with a comment naming the finding). My independent verification, none of it taking the Builder's word:

1. Attack replay from cold — git add -A into a scratch GIT_INDEX_FILE seeded from HEAD: staged paths are main.go only; config.json not staged. git check-ignore -v config.json → .gitignore:7.
2. Fix is on origin, not just local — git show origin/main:.gitignore contains config.json. A fresh clone from origin + dropping the real config.json in → ignored ✅, git add -A does not stage it ✅. This matters: a local-only .gitignore edit would not protect the next clone.
3. Full key never committed — my original evidence used the 6-char prefix and is now contaminated: our own finding/inbox/journal text contains tk_bhg, so -S'tk_bhg' yields false positives. Re-tested against the full 51-char value: git log --all -S"$KEY" → 0 commits in BOTH cc-ci and cc-ci-adv. Binary search on prefix length: the longest prefix ever committed anywhere is 6 of 51 chars, in our own documentation — not a usable disclosure. No leak, past or latent.
4. No non-git exposure — dashboard is live (https://ci.commoninternet.net/ → 200) but /config.json → 404 (also 404 on dashboard.ci.…); no tracked source reads it (the other config.json hits are /root/.docker/config.json, unrelated). Perms -rw-r--r-- loops:users.

CORRECTION to my own finding (Builder was right, I was wrong). I wrote "BOTH Builder clones". There is only one repo: /srv/cc-ci is a symlink → /srv/cc-ci-orch (ls -ld), so /srv/cc-ci/cc-ci and /srv/cc-ci-orch/cc-ci share rev-parse --show-toplevel, the same .git inode (3206558) and the same .gitignore inode (3252849). My cc-ci-adv "pair" is the same illusion. A filesystem-wide sweep found exactly one config.json inside any git repo, and it is now IGNORED. One fix, fully applied — not half.

Residual, explicitly NOT closed by this: the key is still on disk unrotated (len=51, tk_bhg…). Gitignoring prevents a future commit; it cannot un-expose a value that leaked by another channel. Since the full key provably never entered git and is not HTTP-reachable, git is not a reason to rotate. The Builder correctly escalated rotation to the operator rather than deciding it — that judgement was right, and the call remains the operator's.

---

Original report (as filed 2026-07-08T23:12Z)

[adversary] F-redfix-2 — original text — OPEN, NON-BLOCKING

Severity: does NOT block phase redfix (out of scope of its Definition of Done — no VETO, DONE stands). Latent secret-leak risk in the working environment; worth fixing before any future phase does a broad git add.

What: config.json (1128 B, mtime 2026-06-23T00:50Z) exists at the repo root of BOTH Builder clones — /srv/cc-ci/cc-ci and /srv/cc-ci-orch/cc-ci. It holds a live-shaped inference credential at .provider.tinfoil.options.apiKey (51 chars, prefix tk_bhg… — value not reproduced here). The file is untracked, but .gitignore does not cover it: .gitignore lists .testenv, *.key, *.pem, runs/, .claude/ — no config.json. So git check-ignore config.json → miss.

Origin is a real pushed remote (git.autonomic.zone/recipe-maintainers/cc-ci.git, credentials embedded in the remote URL). A single git add -A / git add . in either clone would stage and then push the key.

Good news (verified, not assumed): the key has never been committed — git log --all --oneline -S'tk_bhg' → empty; git log --all -- config.json → empty; git ls-files has no config.json at any path. So this is a latent risk, not an existing leak. The Adversary clones (/srv/cc-ci/cc-ci-adv, /srv/cc-ci-orch/cc-ci-adv) do not carry the file at all.

Repro:

cd /srv/cc-ci/cc-ci && git status --porcelain config.json     # -> "?? config.json"
git check-ignore -v config.json; echo "exit=$?"               # -> exit=1 (NOT ignored)
git log --all --oneline -- config.json                        # -> empty (never committed)

Proposed remedy (Builder — repo change, mine to file, not to make): add config.json to .gitignore under the existing "local secrets / env — never commit" block. Optionally rotate the Tinfoil key if it was ever pasted into a log/transcript. I did not touch, move, or delete the file — it holds a live-looking credential and is not mine to modify.

Discovery: independent break-it probe on my "no secrets in the repo / published logs / dashboard" standing mandate, run after the phase closed. The Builder's journal @418ec57 independently noticed the same file; I verified the exposure surface (gitignore miss + never-committed) from a cold start rather than taking that note at face value. Only the Adversary closes this, after re-test.

[adversary] F-redfix-3 — M2's discourse evidence shas (9ff5e19, 53ba0910) no longer exist on the mirror; the fix content survives — CLOSED @2026-07-08T23:24Z (non-blocking, no VETO)

Severity: does NOT block phase redfix (DONE stands). Evidence-durability defect in the record, not in the fix. Filed so a future auditor of redfix does not conclude "the discourse fix was withdrawn."

What. STATUS-redfix.md pins the discourse fix at 9ff5e19 (fix list) and 53ba0910 (WHERE refs) on recipe-maintainers/discourse branch discourse-official-image. As of 2026-07-08 that branch heads at ede6399 and neither sha resolves: fetching all 17 refs/heads/* + refs/pull/*/head into one clone and running git cat-file -t on each returns not a valid object name. The branch was force-pushed/rebased and extended by later phases (ede6399 = refs/pull/5/head; adds discourse/postgres:pg18 + POSTGRES_USER in pg_backup.sh). redfix's PR is also no longer "#4" — refs/pull/4/head is now 0c4539b7.

Why it is CLOSED rather than a VETO. I re-verified the content the M2 PASS actually asserted, at the current head: compose.yml → image: discourse/discourse:3.5.3 (official-image migration) and compose.smtpauth.yml → 0 sidekiq occurrences (the F-redfix-1 remedy). Both hold at ede6399 and at 0c4539b7. The fix is present and re-verifiable; only the pointers rotted. M2 was correct when given.

Repro. git clone https://git.autonomic.zone/recipe-maintainers/discourse && cd discourse && git fetch origin 'refs/heads/*:refs/remotes/origin/*' 'refs/pull/*/head:refs/remotes/pr/*' && git cat-file -t 9ff5e19 → fatal. Then git show origin/discourse-official-image:compose.yml | grep image: → official image present. (Note: git fetch origin <sha> and a --filter=blob:none clone both give false "absent" signals — use reachability from all refs.)

Lesson for future phases (no action required of the Builder now): shared recipe branches get rewritten, so a sha alone is not durable evidence. Record the content assertion (file → expected line) alongside the sha, or push a tag. The other three redfix fixes pinned exactly (4ca7f418, a0f2db88, 4987ba91), as did cc-ci redfix-m2-harness@07fc6d4a — discourse drifted only because a later phase reused its branch.