cc-ci/machine-docs/STATUS-redfix.md

STATUS — phase redfix

Phase SSOT: /srv/cc-ci/cc-ci-plan/plan-phase-redfix-canon-sweep-failures.md

Mission: investigate every canon-sweep failure (discourse, mattermost-lts, mumble, bluesky-pds, gitea, keycloak) → isolate → root-cause → classify (flake vs genuine; recipe vs test vs warm-machinery vs load) → FIX each (recipe PR or harness improvement) → verify green. No standing exceptions. Nothing merged.

DONE — 2026-07-09T00:18Z

Phase redfix COMPLETE. All six canon-sweep failures investigated in isolation, root-caused, classified, FIXED — each via a recipe PR or a harness improvement — and verified green; no recipe left as a standing exception; nothing merged (operator merges). Every gate has a fresh Adversary PASS in REVIEW-redfix.md and there is no standing VETO:

• M1 PASS @ 2026-06-18T01:18Z (investigation/classification cold-verified).
• M2 PASS @ 2026-07-09T00:18Z (F-redfix-4 remedy cold-verified; VETO CLEARED; supersedes the 2026-06-18T07:06Z M2 PASS, which was given against parent 07fc6d4).

Adversary findings F-redfix-1/2/3/4 are all CLOSED; no open blocking finding.

⚠ OPERATOR ACTION REQUIRED — B-redfix-8 (HIGH, open, outside DoD)

Phase DoD is met, but one HIGH item is open and only an operator can close it. It does not block ## DONE (no DoD item depends on it) and is recorded in full in BACKLOG-redfix.md.

WHAT. The live Gitea bot password (GITEA_PASSWORD in /srv/cc-ci/.testenv) was committed in 14c7dee (machine-docs/BACKLOG-redfix.md), pushed to origin/main, and is served in cleartext to the unauthenticated public internet by the public mirror. It is push-capable to recipe-maintainers/*. HEAD and the redaction commit e99e2b3 are clean; only the historical commit 14c7dee serves it. Redaction cannot unpublish it — the value is permanent in history and in every clone.

WHERE. git.autonomic.zone/recipe-maintainers/cc-ci → /raw/commit/14c7dee/machine-docs/BACKLOG-redfix.md

HOW to confirm exposure (unauthenticated fetch; a raw blob at a fixed sha is immutable, so served size must equal object size):

git cat-file -s 14c7dee:machine-docs/BACKLOG-redfix.md    # → 33408

An unauthenticated fetch of the raw URL returning HTTP 200 / 33408 bytes is the leak, not an error page.

HOW to confirm it is still the live credential (commits to the value without republishing it). Run this on the orchestrator — /srv/cc-ci/.testenv lives there, not on cc-ci:

python3 -c 'import hashlib,re;v=re.search(r"^GITEA_PASSWORD=(.*)$",open("/srv/cc-ci/.testenv").read(),re.M).group(1).strip().strip(chr(34)).strip(chr(39));print(hashlib.sha256(v.encode()).hexdigest()[:16])'

EXPECTED. Prints 3fcea78925015fc9 while still unrotated. A different digest ⇒ rotated ⇒ 14c7dee is inert ⇒ close B-redfix-8 — but only if the probe was sound. Two known-bad probes yield a different digest without any rotation, and must NOT be used to close this item:

• e3b0c44298fc1c14 is the empty-input tell, not a rotation. Running the command over ssh cc-ci fails to open .testenv, hashes the empty string, and prints sha256("")[:16] = e3b0c44298fc1c14 (check: printf '' | sha256sum). Fix the probe; do not close B-redfix-8.
• Do not conflate the three digests. 3fcea78925015fc9 = sha256(credential value)[:16] — the only rotation sentinel. 1994fd8d… = sha256(the whole served blob) — immutable, changes never. e3b0c44298fc1c14 = sha256(empty) — a broken probe.

Exposure confirmed at value level (Adversary, 2026-07-09T07:34Z): the currently-valid, push-capable password appears verbatim in the publicly served body, not merely "a blob that once held a secret."

REMEDY (operator). Rotate the Gitea bot password, then update /srv/cc-ci/.testenv. The credential is a Class-A1 external infra input (plan §4.4) — the Builder must not rotate or invent it. History rewrite is not available to the Builder either (--force is forbidden by the standing rules), so rotation is the only action that actually revokes the exposure.

Merge target: redfix-m2-harness @ b5f2b10 (not 07fc6d4 — that tip carries the F-redfix-4 defect). The earlier ## DONE of 2026-06-18T07:09Z was withdrawn on 2026-07-09 under the standing VETO F-redfix-4 and is re-asserted here only after the remedy was Adversary-verified. Details of that cycle, and the still-open non-blocking B-redfix-5, are below.

One deferred, non-blocking item remains recorded (NOT part of any DoD item, NOT a standing finding): B-redfix-5 in BACKLOG-redfix.md — warm_reconcile.py's rollback warmsnap.restore() sits outside the upgrade's try/except. Pre-dates the enrollment (present at 07fc6d4); F-redfix-4 supplied the only reachable trigger and that is now closed. Adversary concurred it is not a VETO.

---

F-redfix-4 remedy (the reason DONE was withdrawn and re-asserted) — 2026-07-09

WHAT. F-redfix-4 is fixed at redfix-m2-harness @ b5f2b10 (parent 07fc6d4, the sha the M2 PASS was given against). Warm state is now keyed by a stack namespace, not a bare recipe: canonical.canonical_ns(r) is the single source from which BOTH the canonical's domain and its warm-state slot derive. A live-warm provider (r in warm.WARM_DOMAINS) gets ns canon-<r> → domain warm-canon-keycloak.ci.commoninternet.net (unchanged) + slot /var/lib/ci-warm/canon-keycloak/. Every other recipe keeps ns <r> (the invariant is structural — r not in warm.WARM_DOMAINS — not a property of any particular count). WARM_DOMAINS == {"keycloak"} is a singleton, so the re-key is provably keycloak -> canon-keycloak and nothing else: of the 21 enrolled recipes the other 20 satisfy canonical_ns(r) == r, so zero existing canonical slots change. On live disk that is 17 seeded canonicals (those carrying a canonical.json), all of which remain in prune_stale's keep set.

Addresses all four consequences: (1)+(2) slots disjoint, neither deployment can replace the other's known-good; (3) the outage race is removed with the shared slot; (4) prune_stale's reconciler-dir invariant is now structural — <recipe>/ never gains a canonical.json.

Also: warmsnap._assert_slot_not_foreign() refuses to snapshot/restore a slot recorded against a different domain (defence in depth; fails before the destructive swap, not at the next restore). The two comments asserting the deployments "can never touch each other" are corrected (canonical.py, tests/keycloak/recipe_meta.py). WARM_CANONICAL = True is retained — keycloak stays enrolled, no skip-guard, no silent de-enrollment.

MIGRATION: none required. On cc-ci /var/lib/ci-warm/keycloak/ contains only last_good (the canonical was never seeded), so no existing file changes slot. Verify: ls -A /var/lib/ci-warm/keycloak/ → last_good only.

WHERE. cc-ci repo, branch redfix-m2-harness @ b5f2b10. Files: runner/harness/warmsnap.py, runner/harness/canonical.py, runner/warm_reconcile.py, runner/run_recipe_ci.py, tests/keycloak/recipe_meta.py, tests/unit/test_warmsnap.py, tests/unit/test_canonical.py.

HOW to verify (1) — unit suite, cold clone, no docker needed.

git clone -q --branch redfix-m2-harness <cc-ci remote> /tmp/v && cd /tmp/v
/nix/store/x188l04r3gfkh18gy1dpf05fv3kkrgs7-python3-3.12.8-env/bin/python3 -m pytest tests/unit -q

EXPECTED: 325 passed (baseline at parent 07fc6d4 is 315 passed; +10 F-redfix-4 regression tests). The 10 include test_live_and_canonical_slots_are_disjoint, test_slot_maps_1to1_to_its_stack, test_registry_path_of_live_warm_provider_is_not_the_reconciler_dir, test_prune_stale_deenrolled_provider_spares_reconciler_last_good, test_snapshot_refuses_to_clobber_another_domains_slot, test_restore_refuses_a_foreign_slot.

HOW to verify (2) — the VETO's clearing condition, on the real node. Non-destructive: writes only to a scratch CCCI_WARM_ROOT + one throwaway docker volume. Uses the real idle warm-canon-keycloak stack for the canonical side and a throwaway warm-fakelive_…_data volume as the live-warm stand-in (the live warm-keycloak stack cannot be undeployed to snapshot it).

ssh cc-ci
docker volume create warm-fakelive_ci_commoninternet_net_data
MP=$(docker volume inspect -f '{{.Mountpoint}}' warm-fakelive_ci_commoninternet_net_data); echo x > $MP/live.txt
git clone -q --branch redfix-m2-harness <cc-ci remote> /tmp/v && mkdir -p /tmp/vw/keycloak
echo '10.7.1+26.6.2' > /tmp/vw/keycloak/last_good
cd /tmp/v/runner && CCCI_WARM_ROOT=/tmp/vw /nix/store/jag2131a95gw6ng7grig9pj3dn2q8vrv-python3-3.12.8-env/bin/python3 - <<'PY'
import sys; sys.path.insert(0, ".")
from harness import warmsnap as ws, canonical as c
LIVE, CANON = ws.live_slot("keycloak"), c.canonical_slot("keycloak")
CANON_DOM, LIVE_DOM = c.canonical_domain("keycloak"), "warm-fakelive.ci.commoninternet.net"
print(ws.snap_dir(LIVE), ws.snap_dir(CANON), c.registry_path("keycloak"), sep="\n")
ws.snapshot(CANON, CANON_DOM, version="canon-known-good")
ws.snapshot(LIVE,  LIVE_DOM,  version="live-last-good")     # used to clobber the canonical
print("restore(canon) ->", ws.restore(CANON, CANON_DOM)["volumes"])
print("restore(live)  ->", ws.restore(LIVE,  LIVE_DOM )["volumes"])
print("last_good:", open("/tmp/vw/keycloak/last_good").read().strip())
try: ws.snapshot(CANON, LIVE_DOM); print("FAIL: foreign snapshot allowed")
except ws.SnapshotError as e: print("foreign REFUSED:", str(e)[:60])
PY
docker volume rm warm-fakelive_ci_commoninternet_net_data; rm -rf /tmp/vw /tmp/v

EXPECTED (observed 2026-07-09):

/tmp/vw/keycloak/snapshot                     <- live slot
/tmp/vw/canon-keycloak/snapshot               <- canonical slot: DISJOINT
/tmp/vw/canon-keycloak/canonical.json         <- registry NOT in the reconciler's dir
restore(canon) -> ['warm-canon-keycloak_ci_commoninternet_net_mariadb',
                   'warm-canon-keycloak_ci_commoninternet_net_providers']
restore(live)  -> ['warm-fakelive_ci_commoninternet_net_data']
last_good: 10.7.1+26.6.2                      <- survived the canonical seed
foreign REFUSED: warm slot 'canon-keycloak' holds the known-good of ...

Each restore() returns its OWN stack's volumes (the clearing condition). Pre-fix, the same script prints one shared slot /tmp/vw/keycloak/snapshot for both and restore(canon) raises SnapshotError.

Node integrity after my own run of the above (2026-07-09). Real warm root untouched (/var/lib/ci-warm/keycloak/ = last_good only); warm-canon-keycloak mariadb digest byte-identical across the restore round-trip (1201440268 48846 before and after, 391 files / 2 files); live warm-keycloak…/realms/master → 200 throughout; throwaway volume removed; scratch removed.

Scope. keycloak only. M1 classifications and the discourse / mattermost-lts / gitea / bluesky-pds / mumble fixes are untouched by this commit and remain Adversary-verified (M1 PASS @2026-06-18T01:18Z; the five non-keycloak M2 fixes content-verified through re-confirmation #7).

Known residual, NOT in F-redfix-4's clearing condition (filed as B-redfix-5 in BACKLOG-redfix.md, not blocking): warm_reconcile.py's rollback warmsnap.restore() still sits outside the try/except that guards the upgrade, so any restore failure (e.g. a corrupt/absent snapshot) leaves live keycloak undeployed after abra.undeploy(). The F-redfix-4 race that made this reachable is gone; the pre-existing robustness gap is not. Recorded, not silently dropped.

The prior DONE text is retained verbatim below as the historical record of what was claimed on 2026-06-18.

---

(HISTORICAL — withdrawn 2026-07-09 under VETO F-redfix-4; superseded by the DONE above) DONE — 2026-06-18T07:09Z

Phase redfix COMPLETE. All six canon-sweep failures investigated in isolation, root-caused, classified, FIXED — each via a recipe PR or a harness improvement — and verified green; no recipe left as a standing exception; nothing merged (operator merges). Both gates have a fresh Adversary PASS in REVIEW-redfix.md with no standing VETO:

• M1 PASS @ 2026-06-18T01:18Z (investigation/classification cold-verified).
• M2 PASS @ 2026-06-18T07:06Z (all 6 fixes cold-verified; supersedes the 06:42Z FAIL after the discourse F-redfix-1 rework).

Fixes (per recipe): mattermost-lts recipe PR #1 (pg_backup.sh + restore.post-hook) — restore round-trips; discourse recipe PR #4 @9ff5e19 (official-image migration + drop orphaned sidekiq from compose.smtpauth.yml) — level=5, lint R011 ✅; keycloak harness (collision-free warm-canon-<r> + enroll) — promotes without touching live SSO; mumble harness (handshake budget 60→180s) — flake stabilized, non-weakening; gitea recipe PR #2 @a0f2db8 (app.ini seed-on-empty into writable volume) — M1 read-only crash gone; bluesky-pds recipe PR #4 @4987ba9 (caddy ${STACK_NAME}_app) — warm health 200 (was 000). gitea/bluesky end-to-end canonical advance is operator-merge-gated (fix proven by chaos-deploy; published tags don't carry it pre-merge) — consistent with "nothing merged", not a shrug.

Evidence addendum 2026-07-08 (re: F-redfix-3) — discourse sha pins have rotted; verify by CONTENT

The discourse shas cited below (9ff5e19, 53ba0910) no longer resolve on the mirror. A later phase force-pushed and extended the shared branch discourse-official-image (now ede6399 = refs/pull/5/head; redfix's PR is no longer #4). This is branch drift by later work, not a retraction of the redfix fix, and it does not disturb the M2 PASS — which was given against the shas that existed on 2026-06-18. The other three recipe pins and the harness pin are exact and still resolve: mattermost-lts ci/pg-restore@4ca7f418, gitea ci/app-ini-writable@a0f2db88, bluesky-pds ci/warm-routing-alias@4987ba91, cc-ci redfix-m2-harness@07fc6d4a.

Historical sha lines below are left unedited on purpose — they record what was verified when. Use this durable content assertion instead of the shas to re-verify discourse at any future head:

git clone https://git.autonomic.zone/recipe-maintainers/discourse && cd discourse
git show origin/discourse-official-image:compose.yml | grep -m1 'image:.*discourse'
git show origin/discourse-official-image:compose.smtpauth.yml | grep -c sidekiq

EXPECTED: image: discourse/discourse:3.5.3 (the official-image migration = M2's claim) and 0 (the F-redfix-1 orphaned-sidekiq removal). Both re-confirmed by the Builder at ede6399 on 2026-07-08, and by the Adversary at ede6399 and refs/pull/4/head@0c4539b7. (Caveat when reproducing absence of a sha: a --filter=blob:none clone and git fetch origin <sha> both yield false "absent" signals — test reachability from all refs/heads/* + refs/pull/*/head instead.)

---

Phase: M1 — investigate + isolate + classify (IN PROGRESS)

Bootstrapped 2026-06-17T23:20Z. cc-ci healthy, no run in flight, next scheduled sweep 2026-06-21 (3-day clear window). Disk / 38G free (75% used).

Isolation harness (how I reproduce each failure ALONE)

Each canon-sweep per-recipe run is runner/nightly_sweep.run_on_tag(recipe, latest): abra.recipe_checkout(recipe, <latest-tag>) then run_recipe_ci.py with RECIPE=<r> CCCI_SKIP_FETCH=1 and REF/QUICK/MODE/VERSION unset (cold, full, head==tag). Isolation = run ONE recipe at a time with NO concurrent sweep load on the single node (the loaded node is the known flake source per phase plan §2.1). Runs execute on cc-ci from /etc/cc-ci.

Starting canonical state (cc-ci /var/lib/ci-warm/<r>/canonical.json, read 2026-06-17T23:19Z)

Recipe	Canonical now	Note
discourse	(none)	no canonical dir
mattermost-lts	(none)	no canonical dir
mumble	1.0.0+v1.6.870-0 @ 20260617T180501Z	canonical PRESENT, written TODAY — flake signal
bluesky-pds	(none)	no canonical dir
gitea	3.5.3+1.24.2-rootless @ 20260617T083930Z	3.6.0 advance not promoted
keycloak	(none)	de-enrolled (WARM_CANONICAL off)

M1 investigation tracker

Recipe	Isolation run	Result	Root cause	Classification
discourse	DONE @23:40Z (/tmp/redfix-discourse.log on cc-ci)	install/backup/restore/custom PASS; upgrade overlay FAIL. Deploys+serves fine — NOT a timeout/FATA.	cc-ci overlay tests/discourse/test_upgrade.py asserts head runs official discourse/discourse:3.5.3 + drops sidekiq; latest tag 0.8.1+3.5.0 AND main both still bitnamilegacy/discourse:3.5.0+sidekiq (migration exists in no release/main). The depends_on discourse string is a non-fatal prepull-only warning, not the deploy.	stale/PR-specific cc-ci OVERLAY test mismatched to canonical-sweep context (not flake/timeout/recipe-deploy/warm-machinery)
mattermost-lts	DONE @00:05Z (/tmp/redfix-mattermost-lts.log)	install/upgrade/backup/custom PASS; restore FAIL ci_marker does not exist — deterministic in isolation (not a load race)	recipe postgres svc backup labels: backs up hot live PGDATA + dump but has NO backupbot.restore.post-hook to replay the dump → restore doesn't round-trip postgres. Contrast immich (passes): dump-only backup.volumes.postgres.path: backup.sql + restore.post-hook: /pg_backup.sh restore.	genuine RECIPE defect at latest → recipe PR (adopt immich-style dump+restore-post-hook)
mumble	DONE — 2× isolation GREEN (/tmp/redfix-mumble.log + /tmp/redfix-mumble2.log)	ALL tiers PASS incl. handshake on BOTH runs; no orphans; canonical re-promoted green each time	handshake (TLS+ServerSync) not completing within ~60s retry under heavy concurrent sweep load; fine in isolation	load/timing FLAKE → harness stabilization (readiness gate / retry)
bluesky-pds	DONE @00:45Z (/tmp/redfix-bluesky-pds.log + live diag)	cold lifecycle GREEN; WC5 promote 000 reproduces (warm /xrpc/_health last status 0). NOT a flake	caddy on-demand TLS (ask http://app:3000/tls-check) can't reach app: caddy resolves bare app to OTHER stacks' app endpoints on shared proxy net (getent app→only 10.10.0.X, never internal 10.0.3.3; proxy has drone/traefik/keycloak/ccci app aliases) → no cert → 000. Promote machinery correct (refused to write canonical).	genuine routing/RECIPE defect (cross-stack app-alias collision on shared proxy) → recipe PR: unique PDS service name/alias. NOT promote-machinery, NOT flake
gitea	DONE @00:14Z (/tmp/redfix-gitea2.log + live container logs)	cold lifecycle (incl fresh 3.5.3→3.6.0 upgrade) PASS; warm advance crash-loops	LoadCommonSettings() [F] error saving JWT Secret … failed to save "/etc/gitea/app.ini": read-only file system — gitea 3.6.0/1.24.2 tries to persist a JWT to the read-only app.ini docker-config mount on warm reattach (before DB migration; 3.5.3 data intact). Cold passes (fresh secrets, no rewrite).	genuine RECIPE defect (3.6.0 + read-only app.ini config mount on advance) → recipe PR: render app.ini into the writable config volume. (1st gitea run hit a nixenv "already deployed" leftover confound — fixed by undeploying to idle then re-running)
keycloak	DONE @01:05Z (code-verified; no run)	de-enrolled. canonical_domain("keycloak") == WARM_DOMAINS["keycloak"] == warm-keycloak.ci.commoninternet.net EXACTLY (canonical.py:42, warm.py:27,44). Live keycloak 200 /realms/master.	data-warm canonical domain uses same warm-<r> scheme as the live-warm OIDC provider → promote would collide with live shared SSO. No collision-free canonical namespace exists.	HARNESS defect (warm-domain namespace collision) → fix: collision-free canonical_domain for live-warm providers (warm-canon-<r>), then enroll keycloak

M1 results table (recipe → failure → isolation result → root cause → classification → fix approach)

Recipe	Canon-sweep failure	Isolation result	Flake or genuine	Root cause	Class	Fix approach (M2)
discourse	"cold-deploy timeout / deploy FATA"	install/backup/restore/custom GREEN; upgrade overlay RED	genuine (deterministic) — but the canon root-cause was WRONG (no timeout, no deploy FATA)	cc-ci overlay tests/discourse/test_upgrade.py asserts head = official discourse/discourse:3.5.3 + sidekiq dropped; that migration is in NO release tag and NOT in main (all use bitnamilegacy/discourse:3.5.0+sidekiq)	stale/PR-specific cc-ci OVERLAY test	make the overlay assert migration-faithfulness only when the head IS that migration (not vs a release tag), OR a recipe PR migrating off deprecated bitnamilegacy — settle in M2 (NOT a test-weakening)
mattermost-lts	test_restore_returns_state RED	install/upgrade/backup/custom GREEN; restore RED (ci_marker does not exist)	genuine (deterministic in isolation) — NOT the canon "loaded-node race"	recipe postgres backup labels back up hot PGDATA + a dump but have no backupbot.restore.post-hook to replay it; restore doesn't round-trip. immich (passes) uses dump-only path + restore.post-hook	genuine RECIPE defect	recipe PR: adopt immich-style postgres dump + backupbot.restore.post-hook replay
mumble	test_handshake… RED	ALL tiers GREEN in isolation (×N) incl. handshake	FLAKE (load/timing)	handshake (TLS+ServerSync) doesn't complete within the 60s retry under heavy concurrent sweep load; fine isolated; canonical written green today	load/concurrency FLAKE	harness stabilization: stronger readiness gate before the custom tier / longer-or-smarter handshake retry
bluesky-pds	warm promote /xrpc/_health → 000	cold lifecycle GREEN; warm promote 000 reproduces	genuine (deterministic) — NOT a load/rate-limit flake	caddy on-demand TLS calls http://app:3000/tls-check; caddy resolves bare app to OTHER stacks' app endpoints on the shared proxy net (every stack aliases its main svc app), never bluesky's own internal app (10.0.3.3) → connection refused → no cert → 000	genuine ROUTING/RECIPE defect (cross-stack app-alias collision)	recipe PR: give the PDS service a unique name/alias so caddy resolves only bluesky's app
gitea	3.5.3→3.6.0 warm advance doesn't promote	cold (incl fresh upgrade) GREEN; warm advance crash-loops	genuine (deterministic)	gitea 3.6.0/1.24.2 saves a JWT secret to /etc/gitea/app.ini on warm reattach; app.ini is a read-only docker-config mount → read-only file system FATA at LoadCommonSettings (pre-migration; 3.5.3 data intact). Cold passes (fresh secrets, no rewrite)	genuine RECIPE defect	recipe PR: render app.ini into the writable config:/etc/gitea volume (entrypoint) instead of a read-only docker config
keycloak	de-enrolled (not tested)	code-verified (no run)	genuine (structural)	canonical_domain("keycloak") == WARM_DOMAINS["keycloak"] == warm-keycloak.ci.commoninternet.net EXACTLY → a data-warm canonical would collide with the live-warm OIDC provider	HARNESS defect (warm-domain namespace collision)	harness: collision-free canonical_domain for live-warm providers (warm-canon-<r>), then enroll keycloak (WARM_CANONICAL=True)

HOW the Adversary cold-verifies each classification (run ONE recipe at a time, no concurrent load)

Isolation invocation (per recipe R at latest tag T), from /etc/cc-ci on cc-ci: git -C ~/.abra/recipes/R checkout -f --quiet T && env -u REF -u CCCI_QUICK -u MODE -u VERSION RECIPE=R CCCI_SKIP_FETCH=1 cc-ci-run runner/run_recipe_ci.py Latest tags: discourse 0.8.1+3.5.0, mattermost-lts 2.1.9+10.11.15, mumble 1.0.0+v1.6.870-0, bluesky-pds 0.3.0+v0.4.219, gitea 3.6.0+1.24.2-rootless.

• discourse — EXPECT install/backup/restore/custom pass, upgrade fail on test_head_runs_official_image_not_bitnamilegacy + test_sidekiq_service_dropped_by_head. Confirm the overlay mismatch statically: git -C ~/.abra/recipes/discourse show 0.8.1+3.5.0:compose.yml | grep -A1 ' app:' and ... show main:compose.yml both = bitnamilegacy/discourse:3.5.0; grep -c 'sidekiq:' = 1 in both. So the test's discourse/discourse:3.5.3/no-sidekiq expectation exists nowhere upstream.
• mattermost-lts — EXPECT restore fail relation "ci_marker" does not exist. Confirm root cause statically: git -C ~/.abra/recipes/mattermost-lts show 2.1.9+10.11.15:compose.yml | grep backupbot shows pre-hook + backup.path but NO restore.post-hook; immich git -C ~/.abra/recipes/immich show <latest>:compose.yml | grep backupbot shows restore.post-hook: /pg_backup.sh restore.
• mumble — EXPECT all tiers green (run 2–3× to confirm reproducibly green isolated). Canonical written green: cat /var/lib/ci-warm/mumble/canonical.json.
• bluesky-pds — EXPECT cold green, WC5 promote !! WC5 promote failed … warm-bluesky-pds … last status 0. While the warm stack is up, confirm root cause: caddy logs dial tcp 10.10.0.X:3000: connect: connection refused for app:3000/tls-check; docker exec <caddy> getent hosts app returns proxy IPs (10.10.0.X), the app's real internal IP is 10.0.3.x; docker network inspect proxy | grep _app shows many stacks aliasing app. (Tear down the orphaned warm-bluesky-pds stack + volumes after.)
• gitea — REQUIRES idle canonical first: if warm-gitea is deployed, docker stack rm warm-gitea_ci_commoninternet_net (retains data+config volumes) so the advance reattaches from idle. EXPECT cold green, warm advance crash-loop with container log LoadCommonSettings() [F] error saving JWT Secret … "/etc/gitea/app.ini": read-only file system. Restore: leave warm-gitea undeployed (idle 3.5.3, volumes retained) — registry stays 3.5.3+1.24.2-rootless.
• keycloak — no run. Code-verify: canonical.canonical_domain('keycloak') → warm.stable_domain('keycloak') → warm-keycloak.ci.commoninternet.net; warm.WARM_DOMAINS['keycloak'] == same string (runner/harness/canonical.py:42-44, warm.py:27-29,44-48). Live keycloak 200 on /realms/master.

Node state left clean

All isolation runs torn down; orphaned warm-bluesky-pds stack+volumes removed; warm-gitea restored to idle 3.5.3 (volumes retained, registry unchanged); only live warm-keycloak deployed (healthy). No run_recipe_ci.py processes.

M1 — PASS @ 2026-06-18T01:18Z (REVIEW-redfix.md; all 6 classifications cold-verified CORRECT by Adversary's own isolation re-runs). No VETO. Cleared to M2.

Phase: M2 — FIX + verify all six (IN PROGRESS)

Fix designs locked in BACKLOG-redfix.md. Recipe PRs (mattermost-lts/bluesky/gitea) on git.autonomic.zone mirrors via the recipe mirror+PR flow, verified !testme (NEVER merge). Harness fixes (keycloak/mumble) on a cc-ci branch, verified via the harness. discourse: overlay-scope decision. Node now free for my deploys (Adversary done with M1).

M2 fix tracker (updated 2026-06-18T05:53Z — ALL VERIFIED)

Recipe	Class	Fix	PR/branch + ref	Status
mattermost-lts	recipe defect	pg_backup.sh + backupbot.restore.post-hook (immich pattern)	mirror PR #1 ci/pg-restore @4ca7f418	VERIFIED — !testme run #901 ALL tiers green incl test_restore_returns_state
discourse	stale cc-ci overlay	recipe: bitnamilegacy->official discourse image migration + drop orphaned image-less sidekiq from compose.smtpauth.yml (F-redfix-1)	mirror PR #4 discourse-official-image @9ff5e19	VERIFIED — own cold run /tmp/redfix-discourse-m2verify.log level=5 of 5 (all tiers + lint R011 PASS); F-redfix-1 regression fixed
keycloak	harness defect	collision-free canonical_domain (warm-canon-<r> for WARM_DOMAINS recipes) + enroll	cc-ci branch redfix-m2-harness @61211db	VERIFIED — branch-checkout run promotes at warm-canon-keycloak; live warm-keycloak 200 throughout
mumble	load/timing flake	harness: handshake readiness budget 60s->180s	cc-ci branch redfix-m2-harness @07fc6d4	VERIFIED — branch-checkout run all tiers green incl handshake; budget active+non-regressing
gitea	recipe defect	app.ini->staging /etc/gitea/app.ini.init + docker-setup seed-on-EMPTY + DOCKER_SETUP_SH_VERSION v3	mirror PR #2 ci/app-ini-writable @a0f2db8	VERIFIED (direct chaos-deploy; promote merge-gated — see below)
bluesky-pds	recipe defect (routing)	caddy {$APP_HOST}=${STACK_NAME}_app (operator: NO rename) + CADDYFILE_VERSION v2	mirror PR #4 ci/warm-routing-alias @4987ba9	VERIFIED (direct chaos-deploy; promote merge-gated — see below)

cc-ci-side change verification: run from a checkout of redfix-m2-harness (CCCI_REPO=); never touches /etc/cc-ci main. redfix-m2-harness is now mumble+keycloak ONLY (bluesky needs no cc-ci change with the ${STACK_NAME}_app approach; the rename's exec-ref commit b96b8a4 was dropped).

Gate: M2 — RE-CLAIMED, awaiting Adversary (2026-06-18T06:55Z; orig claim 05:53Z)

Re-claim delta (addresses Adversary M2 FAIL @06:42Z — finding F-redfix-1). The first M2 verdict was FAIL on discourse ONLY (other 5 PASS, do-not-redo). F-redfix-1: the official-image migration dropped sidekiq from compose.yml but left a dangling image-less sidekiq: block in compose.smtpauth.yml → L5 lint R011 fail (run level=4) + broken SMTP-auth deploy. FIXED in PR #4 discourse-official-image @9ff5e19 (force-pushed onto @53ba0910): dropped the orphaned sidekiq: block; the app: override already carries DISCOURSE_SMTP_PASSWORD_FILE + smtp_password secret (sidekiq is internal to the official image), so no SMTP coverage lost. grep sidekiq compose*.yml = 0. VERIFIED two ways: (1) the Adversary's exact lint.py repro flow at 9ff5e19 → R011 ✅; (2) my own full cold run /tmp/redfix-discourse-m2verify.log → RUN SUMMARY ... level=5 of 5, all tiers pass (install/upgrade/backup/restore/custom), lint rung: pass. Node clean: no discourse stack, NO discourse canonical (untagged migrated head correctly does not promote — should_promote tagged-gate), recipe reset to published tag 0.8.1+3.5.0. The other 5 fixes are unchanged since their Adversary PASS (keycloak, mumble, gitea, bluesky-pds, mattermost-lts) — no re-run needed.

Adversary cold-verify for discourse: clone discourse @9ff5e19, run RECIPE=discourse CCCI_SKIP_FETCH=1 … run_recipe_ci.py → EXPECT level=5 of 5 (lint R011 ✅, all tiers pass, both upgrade-overlay tests test_head_runs_official_image_not_bitnamilegacy + test_sidekiq_service_dropped_by_head pass); OR the lint-only repro in F-redfix-1 → R011 ✅. grep -c sidekiq ~/.abra/recipes/discourse/compose*.yml @9ff5e19 = 0.

---

Gate: M2 — original claim (2026-06-18T05:53Z)

WHAT (M2 DoD). All six canon-sweep failures FIXED — each via a recipe PR or a harness improvement — and verified green. No recipe left as a standing exception. Nothing merged (operator merges). Per recipe:

• mattermost-lts (recipe PR #1) — added pg_backup.sh + postgres backupbot.restore.post-hook so the logical dump round-trips on restore.
• discourse (recipe PR #4) — migrated the head off deprecated bitnamilegacy to the official discourse/discourse image so the stale PR-faithfulness overlay (test_head_runs_official_image…, test_sidekiq_service_dropped…) passes on the migrated head (NOT a test-weakening).
• keycloak (harness branch) — canonical_domain returns a collision-free warm-canon-<r> for recipes in warm.WARM_DOMAINS (live-warm OIDC providers); keycloak enrolled (WARM_CANONICAL=True).
• mumble (harness branch) — handshake readiness budget widened 60s->180s (load-flake stabilization).
• gitea (recipe PR #2) — app.ini is now seeded into the WRITABLE /etc/gitea volume by docker-setup (if [ ! -s /etc/gitea/app.ini ], seed-on-EMPTY) from the read-only staging config app.ini.init; DOCKER_SETUP_SH_VERSION v1->v3 forces the new docker-setup to re-mount. Gitea 1.24.2 can then persist its JWT secret (the M1 read-only-app.ini crash is gone).
• bluesky-pds (recipe PR #4) — caddy resolves its OWN app via the fully-qualified swarm name ${STACK_NAME}_app (caddy {$APP_HOST} env, set in the caddy service) instead of bare app, which collided with other stacks' app aliases on the shared proxy net. CADDYFILE_VERSION v1->v2.

HOW + EXPECTED + WHERE (Adversary cold-verify, one recipe at a time, no concurrent load):

• mattermost-lts — read-only artifact: /var/lib/cc-ci-runs/901/ on cc-ci — all tiers pass, junit/restore__cc-ci__test_restore.xml testsuite failures=0, test_restore_returns_state pass. OR re-run !testme on PR #1 @4ca7f418. EXPECT restore green.
• discourse — !testme on PR #4 @53ba0910 (run #849 green) OR run from a checkout of the migrated head: EXPECT install/backup/restore/custom + upgrade overlay all pass (head now official image).
• keycloak — from a redfix-m2-harness @61211db checkout (CCCI_REPO=), run RECIPE=keycloak CCCI_SKIP_FETCH=1 ... run_recipe_ci.py. EXPECT all cold tiers pass + WC5 promote succeeds at domain warm-canon-keycloak.ci.commoninternet.net (NOT warm-keycloak); live warm-keycloak.ci.commoninternet.net/realms/master stays 200 throughout. Code: canonical.py canonical_domain returns warm-canon- for r in warm.WARM_DOMAINS.
• mumble — from redfix-m2-harness @07fc6d4 checkout, run RECIPE=mumble CCCI_SKIP_FETCH=1 …. EXPECT all 5 tiers green incl custom/test_protocol_handshake.py::test_handshake_completes_with_ channel_presence; handshake budget = 36 attempts / 180s (was 60s). (Load-flake is not deterministically reproducible; this verifies the stabilization is applied, sound, non-weakening.)
• gitea (recipe PR #2 @a0f2db8 on mirror branch ci/app-ini-writable) — DIRECT chaos-deploy proof (the harness WC5 promote is merge-gated, see NOTE). With the idle 3.5.3 canonical present: cd ~/.abra/recipes/gitea && git checkout -f a0f2db8 then chaos-deploy onto the retained canonical volumes (0-byte app.ini = genuine pre-fix 3.5.3 state): abra app deploy warm-gitea.ci.commoninternet.net -C -o -n. EXPECT: service 1/1; the config volume's app.ini seeded 0->~1862 bytes (INSTALL_LOCK = true); /api/v1/version -> 200 {"version":"1.24.2"} and /api/healthz -> 200 (curl inside the app container); retained 3.5.3 data adopted (data dirs dated 2026-06-17T08:39); ZERO read-only file system crashes in docker service logs (M1 crashed here). Evidence: /tmp/redfix-gitea-m2-directproof.log on cc-ci. Teardown: abra app undeploy … -n, truncate the volume app.ini to 0 (restore pre-fix state). canonical.json stays 3.5.3 idle e6a1cc79.
• bluesky-pds (recipe PR #4 @4987ba9 on mirror branch ci/warm-routing-alias) — DIRECT chaos-deploy proof (warm-promote is the only failing path; merge-gated). git checkout -f 4987ba9; generate secrets (abra app secret generate warm-bluesky-pds.ci.commoninternet.net --all -m -C -o -n) + insert a PLC rotation key (tests/bluesky-pds/install_steps.sh logic: 32-byte hex into pds_plc_rotation_key v1); re-checkout 4987ba9 AFTER secret ops (abra secret insert force-fetches+reverts the checkout); abra app deploy warm-bluesky-pds.ci.commoninternet.net -C -o -n (EXPECT caddyfile: v1 -> v2, NEW DEPLOYMENT 4987ba9). EXPECT: app+caddy 1/1; inside caddy getent hosts warm-bluesky-pds_ci_commoninternet_net_app -> a 10.0.x.x INTERNAL ip (own stack) while getent hosts app -> a 10.10.x.x proxy ip (foreign, the M1 collision); caddy log "certificate obtained successfully" with 0 "connection refused"; external curl https://warm-bluesky-pds.ci. commoninternet.net/xrpc/_health -> 200 {"version":"0.4.219"} (M1 was 000). Evidence: /tmp/redfix-bluesky-m2-directproof.log. Teardown: undeploy + remove volumes (caddy_data, pds_data)
  • secrets (no canonical, matching M1).

NOTE — gitea & bluesky end-to-end canonical-promote is OPERATOR-MERGE-GATED (not a shrug). The harness WC5 promote does a recipe_checkout(published-tag)+non-chaos deploy, and BOTH run_recipe_ci.py:373 AND abra force-fetch refs/tags/* from upstream (abra.py:135 documents this), so any local move of the release tag to the fix commit is reverted to the PUBLISHED commit. The published 3.6.0 / 0.3.0 tags do NOT yet carry the fix (PR not merged — operator merges, per phase guardrail), so pre-merge the promote necessarily deploys the unfixed published release. Confirmed empirically: a full gitea harness run's WC5 promote deployed 357926f and crash-looped exactly like M1. The DIRECT chaos-deploy (chaos = deploy the working-tree checkout = the PR fix) is therefore the MAXIMAL + faithful pre-merge proof — it reproduces the EXACT M1 failing scenario (gitea: the retained canonical volumes; bluesky: warm-bluesky-pds on the shared proxy) and shows the fix resolves it. End-to-end canonical advance follows automatically once the operator merges PR #2 / #4 and the release tag carries the fix. This is NOT a standing exception — the defect is fixed + proven; only the registry-advance awaits the operator's merge (the phase's own "nothing merged" constraint).

WHERE (refs). Recipe PRs on git.autonomic.zone/recipe-maintainers/<recipe>: mattermost-lts ci/pg-restore@4ca7f418, discourse discourse-official-image@53ba0910, gitea ci/app-ini-writable @a0f2db8, bluesky-pds ci/warm-routing-alias@4987ba9. cc-ci harness branch redfix-m2-harness@07fc6d4 (keycloak 61211db + mumble 07fc6d4). Reasoning/dead-ends in JOURNAL-redfix.md. Node left clean (only infra + live warm-keycloak 200; gitea idle 3.5.3 volumes retained, canonical e6a1cc79 unchanged; no bluesky/test stacks/volumes/secrets; no run procs).

Gate: M1 — PASS (above).

WHAT (M1 DoD). All six canon-sweep failures investigated in ISOLATION (one recipe at a time, no concurrent sweep load), root-caused with first-hand evidence, and classified (flake vs genuine; recipe vs test vs warm-machinery vs load) — see the M1 results table + HOW the Adversary cold-verifies sections above. Summary: discourse = stale cc-ci overlay test (canon timeout/FATA root-cause was wrong); mattermost-lts = genuine recipe defect (no backupbot.restore.post-hook); mumble = load/timing FLAKE (2× isolation green); bluesky-pds = genuine routing defect (caddy↔app app-alias collision on shared proxy); gitea = genuine recipe defect (read-only app.ini config mount + 3.6.0 JWT save); keycloak = harness warm-domain namespace collision. NO "probably a flake" — every classification has an isolation re-run or code proof.

HOW + EXPECTED + WHERE. Per-recipe cold-verify commands, expected outputs, and evidence paths are in the two sections above ("M1 results table" and "HOW the Adversary cold-verifies each classification"). Evidence logs on cc-ci: /tmp/redfix-{discourse,mattermost-lts,mumble,mumble2,bluesky-pds,gitea2}.log. Reasoning/dead-ends in JOURNAL-redfix.md. Node left clean (see "Node state left clean" above).

Blocked

(none)