cc-ci/machine-docs/STATUS-redfix.md

STATUS — phase redfix

Phase SSOT: /srv/cc-ci/cc-ci-plan/plan-phase-redfix-canon-sweep-failures.md

Mission: investigate every canon-sweep failure (discourse, mattermost-lts, mumble, bluesky-pds, gitea, keycloak) → isolate → root-cause → classify (flake vs genuine; recipe vs test vs warm-machinery vs load) → FIX each (recipe PR or harness improvement) → verify green. No standing exceptions. Nothing merged.

DONE — 2026-07-09T00:18Z

Phase redfix COMPLETE. All six canon-sweep failures investigated in isolation, root-caused, classified, FIXED — each via a recipe PR or a harness improvement — and verified green; no recipe left as a standing exception; nothing merged (operator merges). Every gate has a fresh Adversary PASS in REVIEW-redfix.md and there is no standing VETO:

• M1 PASS @ 2026-06-18T01:18Z (investigation/classification cold-verified).
• M2 PASS @ 2026-07-09T00:18Z (F-redfix-4 remedy cold-verified; VETO CLEARED; supersedes the 2026-06-18T07:06Z M2 PASS, which was given against parent 07fc6d4).

Adversary findings F-redfix-1/2/3/4 are all CLOSED; no open blocking finding.

⚠ OPERATOR ACTION REQUIRED — B-redfix-8 (HIGH, open, outside DoD)

Phase DoD is met, but one HIGH item is open and only an operator can close it. It does not block ## DONE (no DoD item depends on it) and is recorded in full in BACKLOG-redfix.md.

WHAT. The live Gitea bot password (GITEA_PASSWORD in /srv/cc-ci/.testenv) was committed in 14c7dee (machine-docs/BACKLOG-redfix.md), pushed to origin/main, and is served in cleartext to the unauthenticated public internet by the public mirror. It is push-capable to recipe-maintainers/*. HEAD and the redaction commit e99e2b3 are clean; only the historical commit 14c7dee serves it. Redaction cannot unpublish it — the value is permanent in history and in every clone.

WHERE. git.autonomic.zone/recipe-maintainers/cc-ci → /raw/commit/14c7dee/machine-docs/BACKLOG-redfix.md

HOW to confirm exposure (unauthenticated fetch; a raw blob at a fixed sha is immutable, so served size must equal object size):

git cat-file -s 14c7dee:machine-docs/BACKLOG-redfix.md    # → 33408

An unauthenticated fetch of the raw URL returning HTTP 200 / 33408 bytes is the leak, not an error page.

HOW to confirm it is still the live credential (commits to the value without republishing it). Run this on the orchestrator — /srv/cc-ci/.testenv lives there, not on cc-ci:

python3 -c 'import hashlib,re;v=re.search(r"^GITEA_PASSWORD=(.*)$",open("/srv/cc-ci/.testenv").read(),re.M).group(1).strip().strip(chr(34)).strip(chr(39));print(hashlib.sha256(v.encode()).hexdigest()[:16])'

EXPECTED. Prints 3fcea78925015fc9 while still unrotated. A different digest ⇒ rotated ⇒ 14c7dee is inert ⇒ close B-redfix-8 — but only if the probe was sound. Two known-bad probes yield a different digest without any rotation, and must NOT be used to close this item:

• e3b0c44298fc1c14 is the empty-input tell, not a rotation. Running the command over ssh cc-ci fails to open .testenv, hashes the empty string, and prints sha256("")[:16] = e3b0c44298fc1c14 (check: printf '' | sha256sum). Fix the probe; do not close B-redfix-8.
• Do not conflate the three digests. 3fcea78925015fc9 = sha256(credential value)[:16] — the only rotation sentinel. 1994fd8d… = sha256(the whole served blob) — immutable, changes never. e3b0c44298fc1c14 = sha256(empty) — a broken probe.

Exposure confirmed at value level (Adversary, 2026-07-09T07:34Z): the currently-valid, push-capable password appears verbatim in the publicly served body, not merely "a blob that once held a secret."

REMEDY (operator). Rotate the Gitea bot password, then update /srv/cc-ci/.testenv. The credential is a Class-A1 external infra input (plan §4.4) — the Builder must not rotate or invent it. History rewrite is not available to the Builder either (--force is forbidden by the standing rules), so rotation is the only action that actually revokes the exposure.

Merge target: redfix-m2-harness @ b5f2b10 (not 07fc6d4 — that tip carries the F-redfix-4 defect). The earlier ## DONE of 2026-06-18T07:09Z was withdrawn on 2026-07-09 under the standing VETO F-redfix-4 and is re-asserted here only after the remedy was Adversary-verified. Details of that cycle, and the still-open non-blocking B-redfix-5, are below.

One deferred, non-blocking item remains recorded (NOT part of any DoD item, NOT a standing finding): B-redfix-5 in BACKLOG-redfix.md — in warm_reconcile.py, TWO post-abra.undeploy() calls sit outside the upgrade's try/except: warmsnap.snapshot() on the upgrade path (:514) and warmsnap.restore() on the rollback path (:536). If either raises, live keycloak is left undeployed. Pre-dates the enrollment (present at 07fc6d4). Adversary concurred it is not a VETO.

MERGE PRECONDITION for b5f2b10 (verify at merge time). The F-redfix-4 fix adds a raise path to both of those calls — warmsnap._assert_slot_not_foreign() (warmsnap.py:158 in snapshot(), :209 in restore()). It raises iff the live slot's snapshot/meta.json records a domain other than the one passed. Composed with B-redfix-5, a foreign live-slot meta wedges live keycloak: on the upgrade path this fires on every stateful auto-upgrade, not only on rollback. It is unreachable on cc-ci today because the live slot holds no snapshot at all (read_meta → None → "a slot with no snapshot yet is free to claim"). It is unreachable for that reason — not because F-redfix-4 is closed.

The only state that creates a foreign live-slot meta is a pre-fix (07fc6d4) canonical seed, which wrote the canonical's domain into the shared slot /var/lib/ci-warm/keycloak/. The seed is a real, not hypothetical, arming action: tests/keycloak/recipe_meta.py carries WARM_CANONICAL = True at 07fc6d4 too. So: do not run a canonical keycloak seed from 07fc6d4 between now and the merge.

The wedge leaves keycloak undeployed, and the fix is to delete the foreign snapshot/ — NOT to redeploy. (Corrected 2026-07-09 per Adversary A-redfix-2, which refuted the earlier claim "no code path redeploys it"; the conclusion held but the mechanism was wrong. Re-derived independently — see below.)

reconcile()'s only try/except is warm_reconcile.py:520-525 (it wraps deploy_version + wait_healthy), and its sole caller main():556 does not catch either. A raise at :514 or :536 escapes to SystemExit with live keycloak already undeployed by the preceding abra.undeploy().

A code path does redeploy it. abra.undeploy() (abra.py:295-297) runs only abra app undeploy -n; it does not remove the app .env. So on the next reconcile(), current_version (:286, reads the .env) is still resolvable while is_deployed (:304, reads docker services) is False → the fresh-deploy branch (:471-479) redeploys current or latest. That branch never calls warmsnap, so the foreign meta cannot block it.

What actually prevents self-healing is the absence of a re-trigger. warm-keycloak.service is Type=oneshot, RemainAfterExit=true, with no timer and no Restart=. So the unit does not re-run on its own: keycloak stays down until the next reboot / nixos-rebuild switch, which heals it via the fresh-deploy branch — and then the following reconcile sees current != latest, takes the upgrade path, and re-wedges at :514, because the foreign snapshot/meta.json is still sitting in the slot.

Consequence for the operator: the failure presents as an intermittent flake (alternating wedge / heal across reboots), not a permanent outage — and redeploying keycloak does not fix it. The remediation is to delete the foreign snapshot/ from the slot (or fix B-redfix-5); until that meta is gone, every due upgrade re-wedges.

HOW to verify (all four legs, from a clone at b5f2b10):

git show redfix-m2-harness:runner/harness/abra.py     | sed -n '295,297p'   # undeploy: no .env removal
git show redfix-m2-harness:runner/warm_reconcile.py   | sed -n '471,479p'   # fresh-deploy branch, no warmsnap
git grep -n warm-keycloak -- nix/ | grep -iE 'timer|OnCalendar|Restart='    # → EMPTY (no re-trigger)
ssh cc-ci 'systemctl cat warm-keycloak.service | grep -iE "Type=|RemainAfterExit|Restart="'

EXPECTED: undeploy() body is a single _run(["app","undeploy",domain,"-n"], …) with no .env handling; :472 is if not deployed: returning deployed-fresh:{target} at :479 with no warmsnap call in between; the git grep prints nothing; systemctl cat prints Type=oneshot + RemainAfterExit=true and no Restart= (observed on cc-ci 2026-07-09), and systemctl list-timers --all | grep -i keycloak is empty.

Blast radius: keycloak only. warm.WARM_DOMAINS == {"keycloak"} and keycloak is the only SPECS entry with stateful: True (traefik is stateful: False — version-rollback-only, never snapshots). No other warm unit reaches either call site.

HOW to verify the precondition (on cc-ci; python3 is absent there, so ls/cat):

ssh cc-ci 'ls -A /var/lib/ci-warm/keycloak/; cat /var/lib/ci-warm/keycloak/snapshot/meta.json 2>&1'

EXPECTED (observed 2026-07-09T08:2xZ): last_good only, and cat reports No such file or directory — no snapshot/, hence no meta.json. A meta.json whose "domain" is anything other than warm-keycloak.ci.commoninternet.net means the precondition is violated: fix B-redfix-5 (or delete the foreign slot meta) BEFORE merging b5f2b10. A meta.json recording warm-keycloak.ci.commoninternet.net is self-consistent and passes the guard.

---

F-redfix-4 remedy (the reason DONE was withdrawn and re-asserted) — 2026-07-09

WHAT. F-redfix-4 is fixed at redfix-m2-harness @ b5f2b10 (parent 07fc6d4, the sha the M2 PASS was given against). Warm state is now keyed by a stack namespace, not a bare recipe: canonical.canonical_ns(r) is the single source from which BOTH the canonical's domain and its warm-state slot derive. A live-warm provider (r in warm.WARM_DOMAINS) gets ns canon-<r> → domain warm-canon-keycloak.ci.commoninternet.net (unchanged) + slot /var/lib/ci-warm/canon-keycloak/. Every other recipe keeps ns <r> (the invariant is structural — r not in warm.WARM_DOMAINS — not a property of any particular count). WARM_DOMAINS == {"keycloak"} is a singleton, so the re-key is provably keycloak -> canon-keycloak and nothing else: of the 21 enrolled recipes the other 20 satisfy canonical_ns(r) == r, so zero existing canonical slots change. On live disk that is 17 seeded canonicals (those carrying a canonical.json), all of which remain in prune_stale's keep set.

Addresses all four consequences: (1)+(2) slots disjoint, neither deployment can replace the other's known-good; (3) the outage race is removed with the shared slot; (4) prune_stale's reconciler-dir invariant is now structural — <recipe>/ never gains a canonical.json.

Also: warmsnap._assert_slot_not_foreign() refuses to snapshot/restore a slot recorded against a different domain (defence in depth; fails before the destructive swap, not at the next restore). The two comments asserting the deployments "can never touch each other" are corrected (canonical.py, tests/keycloak/recipe_meta.py). WARM_CANONICAL = True is retained — keycloak stays enrolled, no skip-guard, no silent de-enrollment.

MIGRATION: none required. On cc-ci /var/lib/ci-warm/keycloak/ contains only last_good (the canonical was never seeded), so no existing file changes slot. Verify: ls -A /var/lib/ci-warm/keycloak/ → last_good only.

WHERE. cc-ci repo, branch redfix-m2-harness @ b5f2b10. Files: runner/harness/warmsnap.py, runner/harness/canonical.py, runner/warm_reconcile.py, runner/run_recipe_ci.py, tests/keycloak/recipe_meta.py, tests/unit/test_warmsnap.py, tests/unit/test_canonical.py.

HOW to verify (1) — unit suite, cold clone, no docker needed.

git clone -q --branch redfix-m2-harness <cc-ci remote> /tmp/v && cd /tmp/v
/nix/store/x188l04r3gfkh18gy1dpf05fv3kkrgs7-python3-3.12.8-env/bin/python3 -m pytest tests/unit -q

EXPECTED: 325 passed (baseline at parent 07fc6d4 is 315 passed; +10 F-redfix-4 regression tests). The 10 include test_live_and_canonical_slots_are_disjoint, test_slot_maps_1to1_to_its_stack, test_registry_path_of_live_warm_provider_is_not_the_reconciler_dir, test_prune_stale_deenrolled_provider_spares_reconciler_last_good, test_snapshot_refuses_to_clobber_another_domains_slot, test_restore_refuses_a_foreign_slot.

HOW to verify (2) — the VETO's clearing condition, on the real node. Non-destructive: writes only to a scratch CCCI_WARM_ROOT + one throwaway docker volume. Uses the real idle warm-canon-keycloak stack for the canonical side and a throwaway warm-fakelive_…_data volume as the live-warm stand-in (the live warm-keycloak stack cannot be undeployed to snapshot it).

ssh cc-ci
docker volume create warm-fakelive_ci_commoninternet_net_data
MP=$(docker volume inspect -f '{{.Mountpoint}}' warm-fakelive_ci_commoninternet_net_data); echo x > $MP/live.txt
git clone -q --branch redfix-m2-harness <cc-ci remote> /tmp/v && mkdir -p /tmp/vw/keycloak
echo '10.7.1+26.6.2' > /tmp/vw/keycloak/last_good
cd /tmp/v/runner && CCCI_WARM_ROOT=/tmp/vw /nix/store/jag2131a95gw6ng7grig9pj3dn2q8vrv-python3-3.12.8-env/bin/python3 - <<'PY'
import sys; sys.path.insert(0, ".")
from harness import warmsnap as ws, canonical as c
LIVE, CANON = ws.live_slot("keycloak"), c.canonical_slot("keycloak")
CANON_DOM, LIVE_DOM = c.canonical_domain("keycloak"), "warm-fakelive.ci.commoninternet.net"
print(ws.snap_dir(LIVE), ws.snap_dir(CANON), c.registry_path("keycloak"), sep="\n")
ws.snapshot(CANON, CANON_DOM, version="canon-known-good")
ws.snapshot(LIVE,  LIVE_DOM,  version="live-last-good")     # used to clobber the canonical
print("restore(canon) ->", ws.restore(CANON, CANON_DOM)["volumes"])
print("restore(live)  ->", ws.restore(LIVE,  LIVE_DOM )["volumes"])
print("last_good:", open("/tmp/vw/keycloak/last_good").read().strip())
try: ws.snapshot(CANON, LIVE_DOM); print("FAIL: foreign snapshot allowed")
except ws.SnapshotError as e: print("foreign REFUSED:", str(e)[:60])
PY
docker volume rm warm-fakelive_ci_commoninternet_net_data; rm -rf /tmp/vw /tmp/v

EXPECTED (observed 2026-07-09):

/tmp/vw/keycloak/snapshot                     <- live slot
/tmp/vw/canon-keycloak/snapshot               <- canonical slot: DISJOINT
/tmp/vw/canon-keycloak/canonical.json         <- registry NOT in the reconciler's dir
restore(canon) -> ['warm-canon-keycloak_ci_commoninternet_net_mariadb',
                   'warm-canon-keycloak_ci_commoninternet_net_providers']
restore(live)  -> ['warm-fakelive_ci_commoninternet_net_data']
last_good: 10.7.1+26.6.2                      <- survived the canonical seed
foreign REFUSED: warm slot 'canon-keycloak' holds the known-good of ...

Each restore() returns its OWN stack's volumes (the clearing condition). Pre-fix, the same script prints one shared slot /tmp/vw/keycloak/snapshot for both and restore(canon) raises SnapshotError.

Node integrity after my own run of the above (2026-07-09). Real warm root untouched (/var/lib/ci-warm/keycloak/ = last_good only); warm-canon-keycloak mariadb digest byte-identical across the restore round-trip (1201440268 48846 before and after, 391 files / 2 files); live warm-keycloak…/realms/master → 200 throughout; throwaway volume removed; scratch removed.

Scope. keycloak only. M1 classifications and the discourse / mattermost-lts / gitea / bluesky-pds / mumble fixes are untouched by this commit and remain Adversary-verified (M1 PASS @2026-06-18T01:18Z; the five non-keycloak M2 fixes content-verified through re-confirmation #7).

Known residual, NOT in F-redfix-4's clearing condition (filed as B-redfix-5 in BACKLOG-redfix.md, not blocking): warm_reconcile.py's rollback warmsnap.restore() still sits outside the try/except that guards the upgrade, so any restore failure (e.g. a corrupt/absent snapshot) leaves live keycloak undeployed after abra.undeploy(). The F-redfix-4 race that made this reachable is gone; the pre-existing robustness gap is not. Recorded, not silently dropped.

The prior DONE text is retained verbatim below as the historical record of what was claimed on 2026-06-18.

---

(HISTORICAL — withdrawn 2026-07-09 under VETO F-redfix-4; superseded by the DONE above) DONE — 2026-06-18T07:09Z

Phase redfix COMPLETE. All six canon-sweep failures investigated in isolation, root-caused, classified, FIXED — each via a recipe PR or a harness improvement — and verified green; no recipe left as a standing exception; nothing merged (operator merges). Both gates have a fresh Adversary PASS in REVIEW-redfix.md with no standing VETO:

• M1 PASS @ 2026-06-18T01:18Z (investigation/classification cold-verified).
• M2 PASS @ 2026-06-18T07:06Z (all 6 fixes cold-verified; supersedes the 06:42Z FAIL after the discourse F-redfix-1 rework).

Fixes (per recipe): mattermost-lts recipe PR #1 (pg_backup.sh + restore.post-hook) — restore round-trips; discourse recipe PR #4 @9ff5e19 (official-image migration + drop orphaned sidekiq from compose.smtpauth.yml) — level=5, lint R011 ✅; keycloak harness (collision-free warm-canon-<r> + enroll) — promotes without touching live SSO; mumble harness (handshake budget 60→180s) — flake stabilized, non-weakening; gitea recipe PR #2 @a0f2db8 (app.ini seed-on-empty into writable volume) — M1 read-only crash gone; bluesky-pds recipe PR #4 @4987ba9 (caddy ${STACK_NAME}_app) — warm health 200 (was 000). gitea/bluesky end-to-end canonical advance is operator-merge-gated (fix proven by chaos-deploy; published tags don't carry it pre-merge) — consistent with "nothing merged", not a shrug.

Evidence addendum 2026-07-08 (re: F-redfix-3) — discourse sha pins have rotted; verify by CONTENT

The discourse shas cited below (9ff5e19, 53ba0910) no longer resolve on the mirror. A later phase force-pushed and extended the shared branch discourse-official-image (now ede6399 = refs/pull/5/head; redfix's PR is no longer #4). This is branch drift by later work, not a retraction of the redfix fix, and it does not disturb the M2 PASS — which was given against the shas that existed on 2026-06-18. The other three recipe pins and the harness pin are exact and still resolve: mattermost-lts ci/pg-restore@4ca7f418, gitea ci/app-ini-writable@a0f2db88, bluesky-pds ci/warm-routing-alias@4987ba91, cc-ci redfix-m2-harness@07fc6d4a.

Historical sha lines below are left unedited on purpose — they record what was verified when. Use this durable content assertion instead of the shas to re-verify discourse at any future head:

git clone https://git.autonomic.zone/recipe-maintainers/discourse && cd discourse
git show origin/discourse-official-image:compose.yml | grep -m1 'image:.*discourse'
git show origin/discourse-official-image:compose.smtpauth.yml | grep -c sidekiq

EXPECTED: image: discourse/discourse:3.5.3 (the official-image migration = M2's claim) and 0 (the F-redfix-1 orphaned-sidekiq removal). Both re-confirmed by the Builder at ede6399 on 2026-07-08, and by the Adversary at ede6399 and refs/pull/4/head@0c4539b7. (Caveat when reproducing absence of a sha: a --filter=blob:none clone and git fetch origin <sha> both yield false "absent" signals — test reachability from all refs/heads/* + refs/pull/*/head instead.)

---

Phase: M1 — investigate + isolate + classify (IN PROGRESS)

Bootstrapped 2026-06-17T23:20Z. cc-ci healthy, no run in flight, next scheduled sweep 2026-06-21 (3-day clear window). Disk / 38G free (75% used).

Isolation harness (how I reproduce each failure ALONE)

Each canon-sweep per-recipe run is runner/nightly_sweep.run_on_tag(recipe, latest): abra.recipe_checkout(recipe, <latest-tag>) then run_recipe_ci.py with RECIPE=<r> CCCI_SKIP_FETCH=1 and REF/QUICK/MODE/VERSION unset (cold, full, head==tag). Isolation = run ONE recipe at a time with NO concurrent sweep load on the single node (the loaded node is the known flake source per phase plan §2.1). Runs execute on cc-ci from /etc/cc-ci.

Starting canonical state (cc-ci /var/lib/ci-warm/<r>/canonical.json, read 2026-06-17T23:19Z)

Recipe	Canonical now	Note
discourse	(none)	no canonical dir
mattermost-lts	(none)	no canonical dir
mumble	1.0.0+v1.6.870-0 @ 20260617T180501Z	canonical PRESENT, written TODAY — flake signal
bluesky-pds	(none)	no canonical dir
gitea	3.5.3+1.24.2-rootless @ 20260617T083930Z	3.6.0 advance not promoted
keycloak	(none)	de-enrolled (WARM_CANONICAL off)

M1 investigation tracker

Recipe	Isolation run	Result	Root cause	Classification
discourse	DONE @23:40Z (/tmp/redfix-discourse.log on cc-ci)	install/backup/restore/custom PASS; upgrade overlay FAIL. Deploys+serves fine — NOT a timeout/FATA.	cc-ci overlay tests/discourse/test_upgrade.py asserts head runs official discourse/discourse:3.5.3 + drops sidekiq; latest tag 0.8.1+3.5.0 AND main both still bitnamilegacy/discourse:3.5.0+sidekiq (migration exists in no release/main). The depends_on discourse string is a non-fatal prepull-only warning, not the deploy.	stale/PR-specific cc-ci OVERLAY test mismatched to canonical-sweep context (not flake/timeout/recipe-deploy/warm-machinery)
mattermost-lts	DONE @00:05Z (/tmp/redfix-mattermost-lts.log)	install/upgrade/backup/custom PASS; restore FAIL ci_marker does not exist — deterministic in isolation (not a load race)	recipe postgres svc backup labels: backs up hot live PGDATA + dump but has NO backupbot.restore.post-hook to replay the dump → restore doesn't round-trip postgres. Contrast immich (passes): dump-only backup.volumes.postgres.path: backup.sql + restore.post-hook: /pg_backup.sh restore.	genuine RECIPE defect at latest → recipe PR (adopt immich-style dump+restore-post-hook)
mumble	DONE — 2× isolation GREEN (/tmp/redfix-mumble.log + /tmp/redfix-mumble2.log)	ALL tiers PASS incl. handshake on BOTH runs; no orphans; canonical re-promoted green each time	handshake (TLS+ServerSync) not completing within ~60s retry under heavy concurrent sweep load; fine in isolation	load/timing FLAKE → harness stabilization (readiness gate / retry)
bluesky-pds	DONE @00:45Z (/tmp/redfix-bluesky-pds.log + live diag)	cold lifecycle GREEN; WC5 promote 000 reproduces (warm /xrpc/_health last status 0). NOT a flake	caddy on-demand TLS (ask http://app:3000/tls-check) can't reach app: caddy resolves bare app to OTHER stacks' app endpoints on shared proxy net (getent app→only 10.10.0.X, never internal 10.0.3.3; proxy has drone/traefik/keycloak/ccci app aliases) → no cert → 000. Promote machinery correct (refused to write canonical).	genuine routing/RECIPE defect (cross-stack app-alias collision on shared proxy) → recipe PR: unique PDS service name/alias. NOT promote-machinery, NOT flake
gitea	DONE @00:14Z (/tmp/redfix-gitea2.log + live container logs)	cold lifecycle (incl fresh 3.5.3→3.6.0 upgrade) PASS; warm advance crash-loops	LoadCommonSettings() [F] error saving JWT Secret … failed to save "/etc/gitea/app.ini": read-only file system — gitea 3.6.0/1.24.2 tries to persist a JWT to the read-only app.ini docker-config mount on warm reattach (before DB migration; 3.5.3 data intact). Cold passes (fresh secrets, no rewrite).	genuine RECIPE defect (3.6.0 + read-only app.ini config mount on advance) → recipe PR: render app.ini into the writable config volume. (1st gitea run hit a nixenv "already deployed" leftover confound — fixed by undeploying to idle then re-running)
keycloak	DONE @01:05Z (code-verified; no run)	de-enrolled. canonical_domain("keycloak") == WARM_DOMAINS["keycloak"] == warm-keycloak.ci.commoninternet.net EXACTLY (canonical.py:42, warm.py:27,44). Live keycloak 200 /realms/master.	data-warm canonical domain uses same warm-<r> scheme as the live-warm OIDC provider → promote would collide with live shared SSO. No collision-free canonical namespace exists.	HARNESS defect (warm-domain namespace collision) → fix: collision-free canonical_domain for live-warm providers (warm-canon-<r>), then enroll keycloak

M1 results table (recipe → failure → isolation result → root cause → classification → fix approach)

Recipe	Canon-sweep failure	Isolation result	Flake or genuine	Root cause	Class	Fix approach (M2)
discourse	"cold-deploy timeout / deploy FATA"	install/backup/restore/custom GREEN; upgrade overlay RED	genuine (deterministic) — but the canon root-cause was WRONG (no timeout, no deploy FATA)	cc-ci overlay tests/discourse/test_upgrade.py asserts head = official discourse/discourse:3.5.3 + sidekiq dropped; that migration is in NO release tag and NOT in main (all use bitnamilegacy/discourse:3.5.0+sidekiq)	stale/PR-specific cc-ci OVERLAY test	make the overlay assert migration-faithfulness only when the head IS that migration (not vs a release tag), OR a recipe PR migrating off deprecated bitnamilegacy — settle in M2 (NOT a test-weakening)
mattermost-lts	test_restore_returns_state RED	install/upgrade/backup/custom GREEN; restore RED (ci_marker does not exist)	genuine (deterministic in isolation) — NOT the canon "loaded-node race"	recipe postgres backup labels back up hot PGDATA + a dump but have no backupbot.restore.post-hook to replay it; restore doesn't round-trip. immich (passes) uses dump-only path + restore.post-hook	genuine RECIPE defect	recipe PR: adopt immich-style postgres dump + backupbot.restore.post-hook replay
mumble	test_handshake… RED	ALL tiers GREEN in isolation (×N) incl. handshake	FLAKE (load/timing)	handshake (TLS+ServerSync) doesn't complete within the 60s retry under heavy concurrent sweep load; fine isolated; canonical written green today	load/concurrency FLAKE	harness stabilization: stronger readiness gate before the custom tier / longer-or-smarter handshake retry
bluesky-pds	warm promote /xrpc/_health → 000	cold lifecycle GREEN; warm promote 000 reproduces	genuine (deterministic) — NOT a load/rate-limit flake	caddy on-demand TLS calls http://app:3000/tls-check; caddy resolves bare app to OTHER stacks' app endpoints on the shared proxy net (every stack aliases its main svc app), never bluesky's own internal app (10.0.3.3) → connection refused → no cert → 000	genuine ROUTING/RECIPE defect (cross-stack app-alias collision)	recipe PR: give the PDS service a unique name/alias so caddy resolves only bluesky's app
gitea	3.5.3→3.6.0 warm advance doesn't promote	cold (incl fresh upgrade) GREEN; warm advance crash-loops	genuine (deterministic)	gitea 3.6.0/1.24.2 saves a JWT secret to /etc/gitea/app.ini on warm reattach; app.ini is a read-only docker-config mount → read-only file system FATA at LoadCommonSettings (pre-migration; 3.5.3 data intact). Cold passes (fresh secrets, no rewrite)	genuine RECIPE defect	recipe PR: render app.ini into the writable config:/etc/gitea volume (entrypoint) instead of a read-only docker config
keycloak	de-enrolled (not tested)	code-verified (no run)	genuine (structural)	canonical_domain("keycloak") == WARM_DOMAINS["keycloak"] == warm-keycloak.ci.commoninternet.net EXACTLY → a data-warm canonical would collide with the live-warm OIDC provider	HARNESS defect (warm-domain namespace collision)	harness: collision-free canonical_domain for live-warm providers (warm-canon-<r>), then enroll keycloak (WARM_CANONICAL=True)

HOW the Adversary cold-verifies each classification (run ONE recipe at a time, no concurrent load)

Isolation invocation (per recipe R at latest tag T), from /etc/cc-ci on cc-ci: git -C ~/.abra/recipes/R checkout -f --quiet T && env -u REF -u CCCI_QUICK -u MODE -u VERSION RECIPE=R CCCI_SKIP_FETCH=1 cc-ci-run runner/run_recipe_ci.py Latest tags: discourse 0.8.1+3.5.0, mattermost-lts 2.1.9+10.11.15, mumble 1.0.0+v1.6.870-0, bluesky-pds 0.3.0+v0.4.219, gitea 3.6.0+1.24.2-rootless.

• discourse — EXPECT install/backup/restore/custom pass, upgrade fail on test_head_runs_official_image_not_bitnamilegacy + test_sidekiq_service_dropped_by_head. Confirm the overlay mismatch statically: git -C ~/.abra/recipes/discourse show 0.8.1+3.5.0:compose.yml | grep -A1 ' app:' and ... show main:compose.yml both = bitnamilegacy/discourse:3.5.0; grep -c 'sidekiq:' = 1 in both. So the test's discourse/discourse:3.5.3/no-sidekiq expectation exists nowhere upstream.
• mattermost-lts — EXPECT restore fail relation "ci_marker" does not exist. Confirm root cause statically: git -C ~/.abra/recipes/mattermost-lts show 2.1.9+10.11.15:compose.yml | grep backupbot shows pre-hook + backup.path but NO restore.post-hook; immich git -C ~/.abra/recipes/immich show <latest>:compose.yml | grep backupbot shows restore.post-hook: /pg_backup.sh restore.
• mumble — EXPECT all tiers green (run 2–3× to confirm reproducibly green isolated). Canonical written green: cat /var/lib/ci-warm/mumble/canonical.json.
• bluesky-pds — EXPECT cold green, WC5 promote !! WC5 promote failed … warm-bluesky-pds … last status 0. While the warm stack is up, confirm root cause: caddy logs dial tcp 10.10.0.X:3000: connect: connection refused for app:3000/tls-check; docker exec <caddy> getent hosts app returns proxy IPs (10.10.0.X), the app's real internal IP is 10.0.3.x; docker network inspect proxy | grep _app shows many stacks aliasing app. (Tear down the orphaned warm-bluesky-pds stack + volumes after.)
• gitea — REQUIRES idle canonical first: if warm-gitea is deployed, docker stack rm warm-gitea_ci_commoninternet_net (retains data+config volumes) so the advance reattaches from idle. EXPECT cold green, warm advance crash-loop with container log LoadCommonSettings() [F] error saving JWT Secret … "/etc/gitea/app.ini": read-only file system. Restore: leave warm-gitea undeployed (idle 3.5.3, volumes retained) — registry stays 3.5.3+1.24.2-rootless.
• keycloak — no run. Code-verify: canonical.canonical_domain('keycloak') → warm.stable_domain('keycloak') → warm-keycloak.ci.commoninternet.net; warm.WARM_DOMAINS['keycloak'] == same string (runner/harness/canonical.py:42-44, warm.py:27-29,44-48). Live keycloak 200 on /realms/master.

Node state left clean

All isolation runs torn down; orphaned warm-bluesky-pds stack+volumes removed; warm-gitea restored to idle 3.5.3 (volumes retained, registry unchanged); only live warm-keycloak deployed (healthy). No run_recipe_ci.py processes.

M1 — PASS @ 2026-06-18T01:18Z (REVIEW-redfix.md; all 6 classifications cold-verified CORRECT by Adversary's own isolation re-runs). No VETO. Cleared to M2.

Phase: M2 — FIX + verify all six (IN PROGRESS)

Fix designs locked in BACKLOG-redfix.md. Recipe PRs (mattermost-lts/bluesky/gitea) on git.autonomic.zone mirrors via the recipe mirror+PR flow, verified !testme (NEVER merge). Harness fixes (keycloak/mumble) on a cc-ci branch, verified via the harness. discourse: overlay-scope decision. Node now free for my deploys (Adversary done with M1).

M2 fix tracker (updated 2026-06-18T05:53Z — ALL VERIFIED)

Recipe	Class	Fix	PR/branch + ref	Status
mattermost-lts	recipe defect	pg_backup.sh + backupbot.restore.post-hook (immich pattern)	mirror PR #1 ci/pg-restore @4ca7f418	VERIFIED — !testme run #901 ALL tiers green incl test_restore_returns_state
discourse	stale cc-ci overlay	recipe: bitnamilegacy->official discourse image migration + drop orphaned image-less sidekiq from compose.smtpauth.yml (F-redfix-1)	mirror PR #4 discourse-official-image @9ff5e19	VERIFIED — own cold run /tmp/redfix-discourse-m2verify.log level=5 of 5 (all tiers + lint R011 PASS); F-redfix-1 regression fixed
keycloak	harness defect	collision-free canonical_domain (warm-canon-<r> for WARM_DOMAINS recipes) + enroll	cc-ci branch redfix-m2-harness @61211db	VERIFIED — branch-checkout run promotes at warm-canon-keycloak; live warm-keycloak 200 throughout
mumble	load/timing flake	harness: handshake readiness budget 60s->180s	cc-ci branch redfix-m2-harness @07fc6d4	VERIFIED — branch-checkout run all tiers green incl handshake; budget active+non-regressing
gitea	recipe defect	app.ini->staging /etc/gitea/app.ini.init + docker-setup seed-on-EMPTY + DOCKER_SETUP_SH_VERSION v3	mirror PR #2 ci/app-ini-writable @a0f2db8	VERIFIED (direct chaos-deploy; promote merge-gated — see below)
bluesky-pds	recipe defect (routing)	caddy {$APP_HOST}=${STACK_NAME}_app (operator: NO rename) + CADDYFILE_VERSION v2	mirror PR #4 ci/warm-routing-alias @4987ba9	VERIFIED (direct chaos-deploy; promote merge-gated — see below)

cc-ci-side change verification: run from a checkout of redfix-m2-harness (CCCI_REPO=); never touches /etc/cc-ci main. redfix-m2-harness is now mumble+keycloak ONLY (bluesky needs no cc-ci change with the ${STACK_NAME}_app approach; the rename's exec-ref commit b96b8a4 was dropped).

Gate: M2 — RE-CLAIMED, awaiting Adversary (2026-06-18T06:55Z; orig claim 05:53Z)

Re-claim delta (addresses Adversary M2 FAIL @06:42Z — finding F-redfix-1). The first M2 verdict was FAIL on discourse ONLY (other 5 PASS, do-not-redo). F-redfix-1: the official-image migration dropped sidekiq from compose.yml but left a dangling image-less sidekiq: block in compose.smtpauth.yml → L5 lint R011 fail (run level=4) + broken SMTP-auth deploy. FIXED in PR #4 discourse-official-image @9ff5e19 (force-pushed onto @53ba0910): dropped the orphaned sidekiq: block; the app: override already carries DISCOURSE_SMTP_PASSWORD_FILE + smtp_password secret (sidekiq is internal to the official image), so no SMTP coverage lost. grep sidekiq compose*.yml = 0. VERIFIED two ways: (1) the Adversary's exact lint.py repro flow at 9ff5e19 → R011 ✅; (2) my own full cold run /tmp/redfix-discourse-m2verify.log → RUN SUMMARY ... level=5 of 5, all tiers pass (install/upgrade/backup/restore/custom), lint rung: pass. Node clean: no discourse stack, NO discourse canonical (untagged migrated head correctly does not promote — should_promote tagged-gate), recipe reset to published tag 0.8.1+3.5.0. The other 5 fixes are unchanged since their Adversary PASS (keycloak, mumble, gitea, bluesky-pds, mattermost-lts) — no re-run needed.

Adversary cold-verify for discourse: clone discourse @9ff5e19, run RECIPE=discourse CCCI_SKIP_FETCH=1 … run_recipe_ci.py → EXPECT level=5 of 5 (lint R011 ✅, all tiers pass, both upgrade-overlay tests test_head_runs_official_image_not_bitnamilegacy + test_sidekiq_service_dropped_by_head pass); OR the lint-only repro in F-redfix-1 → R011 ✅. grep -c sidekiq ~/.abra/recipes/discourse/compose*.yml @9ff5e19 = 0.

---

Gate: M2 — original claim (2026-06-18T05:53Z)

WHAT (M2 DoD). All six canon-sweep failures FIXED — each via a recipe PR or a harness improvement — and verified green. No recipe left as a standing exception. Nothing merged (operator merges). Per recipe:

• mattermost-lts (recipe PR #1) — added pg_backup.sh + postgres backupbot.restore.post-hook so the logical dump round-trips on restore.
• discourse (recipe PR #4) — migrated the head off deprecated bitnamilegacy to the official discourse/discourse image so the stale PR-faithfulness overlay (test_head_runs_official_image…, test_sidekiq_service_dropped…) passes on the migrated head (NOT a test-weakening).
• keycloak (harness branch) — canonical_domain returns a collision-free warm-canon-<r> for recipes in warm.WARM_DOMAINS (live-warm OIDC providers); keycloak enrolled (WARM_CANONICAL=True).
• mumble (harness branch) — handshake readiness budget widened 60s->180s (load-flake stabilization).
• gitea (recipe PR #2) — app.ini is now seeded into the WRITABLE /etc/gitea volume by docker-setup (if [ ! -s /etc/gitea/app.ini ], seed-on-EMPTY) from the read-only staging config app.ini.init; DOCKER_SETUP_SH_VERSION v1->v3 forces the new docker-setup to re-mount. Gitea 1.24.2 can then persist its JWT secret (the M1 read-only-app.ini crash is gone).
• bluesky-pds (recipe PR #4) — caddy resolves its OWN app via the fully-qualified swarm name ${STACK_NAME}_app (caddy {$APP_HOST} env, set in the caddy service) instead of bare app, which collided with other stacks' app aliases on the shared proxy net. CADDYFILE_VERSION v1->v2.

HOW + EXPECTED + WHERE (Adversary cold-verify, one recipe at a time, no concurrent load):

• mattermost-lts — read-only artifact: /var/lib/cc-ci-runs/901/ on cc-ci — all tiers pass, junit/restore__cc-ci__test_restore.xml testsuite failures=0, test_restore_returns_state pass. OR re-run !testme on PR #1 @4ca7f418. EXPECT restore green.
• discourse — !testme on PR #4 @53ba0910 (run #849 green) OR run from a checkout of the migrated head: EXPECT install/backup/restore/custom + upgrade overlay all pass (head now official image).
• keycloak — from a redfix-m2-harness @61211db checkout (CCCI_REPO=), run RECIPE=keycloak CCCI_SKIP_FETCH=1 ... run_recipe_ci.py. EXPECT all cold tiers pass + WC5 promote succeeds at domain warm-canon-keycloak.ci.commoninternet.net (NOT warm-keycloak); live warm-keycloak.ci.commoninternet.net/realms/master stays 200 throughout. Code: canonical.py canonical_domain returns warm-canon- for r in warm.WARM_DOMAINS.
• mumble — from redfix-m2-harness @07fc6d4 checkout, run RECIPE=mumble CCCI_SKIP_FETCH=1 …. EXPECT all 5 tiers green incl custom/test_protocol_handshake.py::test_handshake_completes_with_ channel_presence; handshake budget = 36 attempts / 180s (was 60s). (Load-flake is not deterministically reproducible; this verifies the stabilization is applied, sound, non-weakening.)
• gitea (recipe PR #2 @a0f2db8 on mirror branch ci/app-ini-writable) — DIRECT chaos-deploy proof (the harness WC5 promote is merge-gated, see NOTE). With the idle 3.5.3 canonical present: cd ~/.abra/recipes/gitea && git checkout -f a0f2db8 then chaos-deploy onto the retained canonical volumes (0-byte app.ini = genuine pre-fix 3.5.3 state): abra app deploy warm-gitea.ci.commoninternet.net -C -o -n. EXPECT: service 1/1; the config volume's app.ini seeded 0->~1862 bytes (INSTALL_LOCK = true); /api/v1/version -> 200 {"version":"1.24.2"} and /api/healthz -> 200 (curl inside the app container); retained 3.5.3 data adopted (data dirs dated 2026-06-17T08:39); ZERO read-only file system crashes in docker service logs (M1 crashed here). Evidence: /tmp/redfix-gitea-m2-directproof.log on cc-ci. Teardown: abra app undeploy … -n, truncate the volume app.ini to 0 (restore pre-fix state). canonical.json stays 3.5.3 idle e6a1cc79.
• bluesky-pds (recipe PR #4 @4987ba9 on mirror branch ci/warm-routing-alias) — DIRECT chaos-deploy proof (warm-promote is the only failing path; merge-gated). git checkout -f 4987ba9; generate secrets (abra app secret generate warm-bluesky-pds.ci.commoninternet.net --all -m -C -o -n) + insert a PLC rotation key (tests/bluesky-pds/install_steps.sh logic: 32-byte hex into pds_plc_rotation_key v1); re-checkout 4987ba9 AFTER secret ops (abra secret insert force-fetches+reverts the checkout); abra app deploy warm-bluesky-pds.ci.commoninternet.net -C -o -n (EXPECT caddyfile: v1 -> v2, NEW DEPLOYMENT 4987ba9). EXPECT: app+caddy 1/1; inside caddy getent hosts warm-bluesky-pds_ci_commoninternet_net_app -> a 10.0.x.x INTERNAL ip (own stack) while getent hosts app -> a 10.10.x.x proxy ip (foreign, the M1 collision); caddy log "certificate obtained successfully" with 0 "connection refused"; external curl https://warm-bluesky-pds.ci. commoninternet.net/xrpc/_health -> 200 {"version":"0.4.219"} (M1 was 000). Evidence: /tmp/redfix-bluesky-m2-directproof.log. Teardown: undeploy + remove volumes (caddy_data, pds_data)
  • secrets (no canonical, matching M1).

NOTE — gitea & bluesky end-to-end canonical-promote is OPERATOR-MERGE-GATED (not a shrug). The harness WC5 promote does a recipe_checkout(published-tag)+non-chaos deploy, and BOTH run_recipe_ci.py:373 AND abra force-fetch refs/tags/* from upstream (abra.py:135 documents this), so any local move of the release tag to the fix commit is reverted to the PUBLISHED commit. The published 3.6.0 / 0.3.0 tags do NOT yet carry the fix (PR not merged — operator merges, per phase guardrail), so pre-merge the promote necessarily deploys the unfixed published release. Confirmed empirically: a full gitea harness run's WC5 promote deployed 357926f and crash-looped exactly like M1. The DIRECT chaos-deploy (chaos = deploy the working-tree checkout = the PR fix) is therefore the MAXIMAL + faithful pre-merge proof — it reproduces the EXACT M1 failing scenario (gitea: the retained canonical volumes; bluesky: warm-bluesky-pds on the shared proxy) and shows the fix resolves it. End-to-end canonical advance follows automatically once the operator merges PR #2 / #4 and the release tag carries the fix. This is NOT a standing exception — the defect is fixed + proven; only the registry-advance awaits the operator's merge (the phase's own "nothing merged" constraint).

WHERE (refs). Recipe PRs on git.autonomic.zone/recipe-maintainers/<recipe>: mattermost-lts ci/pg-restore@4ca7f418, discourse discourse-official-image@53ba0910, gitea ci/app-ini-writable @a0f2db8, bluesky-pds ci/warm-routing-alias@4987ba9. cc-ci harness branch redfix-m2-harness@07fc6d4 (keycloak 61211db + mumble 07fc6d4). Reasoning/dead-ends in JOURNAL-redfix.md. Node left clean (only infra + live warm-keycloak 200; gitea idle 3.5.3 volumes retained, canonical e6a1cc79 unchanged; no bluesky/test stacks/volumes/secrets; no run procs).

Gate: M1 — PASS (above).

WHAT (M1 DoD). All six canon-sweep failures investigated in ISOLATION (one recipe at a time, no concurrent sweep load), root-caused with first-hand evidence, and classified (flake vs genuine; recipe vs test vs warm-machinery vs load) — see the M1 results table + HOW the Adversary cold-verifies sections above. Summary: discourse = stale cc-ci overlay test (canon timeout/FATA root-cause was wrong); mattermost-lts = genuine recipe defect (no backupbot.restore.post-hook); mumble = load/timing FLAKE (2× isolation green); bluesky-pds = genuine routing defect (caddy↔app app-alias collision on shared proxy); gitea = genuine recipe defect (read-only app.ini config mount + 3.6.0 JWT save); keycloak = harness warm-domain namespace collision. NO "probably a flake" — every classification has an isolation re-run or code proof.

HOW + EXPECTED + WHERE. Per-recipe cold-verify commands, expected outputs, and evidence paths are in the two sections above ("M1 results table" and "HOW the Adversary cold-verifies each classification"). Evidence logs on cc-ci: /tmp/redfix-{discourse,mattermost-lts,mumble,mumble2,bluesky-pds,gitea2}.log. Reasoning/dead-ends in JOURNAL-redfix.md. Node left clean (see "Node state left clean" above).

Untracked file main.go — NOT produced by this phase (operator: unexplained)

WHAT. An untracked main.go (281 bytes, a Go "Hello, World!" net/http server on :8080) sits in the repo root. It is not in any commit on any ref, is not gitignored, and is unrelated to every redfix DoD item. No go.mod accompanies it. I did not create it and have not committed, modified, or deleted it.

WHERE. /srv/cc-ci-orch/cc-ci/main.go — one file in one clone.

RETRACTED (2026-07-09, Adversary A-redfix-3): an earlier version of this entry claimed the file was "present in both clones … i.e. written by something outside git." That evidence is void. /srv/cc-ci is a symlink to /srv/cc-ci-orch, so /srv/cc-ci/cc-ci/main.go and /srv/cc-ci-orch/cc-ci/main.go are the same inode — one file seen twice, not two clones. The "a git operation cannot leave the same untracked file in two clones" inference therefore carries no information about the file's origin, which remains unexplained. Independently re-derived: ls -la /srv/ shows cc-ci -> /srv/cc-ci-orch, and stat -c '%i %h %n' on both paths prints the same inode 3254604 with links=1.

HOW to verify. git log --all --oneline -- main.go → empty (in no commit). git status --short → ?? main.go. ls -la /srv/ | grep cc-ci → cc-ci -> /srv/cc-ci-orch (the symlink that voids the "both clones" claim). stat -c '%i %h %n' /srv/cc-ci/cc-ci/main.go /srv/cc-ci-orch/cc-ci/main.go → same inode, links=1.

Risk: inert (Adversary-confirmed). 281-byte hello-world net/http listener; no go.mod; go is not installed; nothing listening on :8080; in no commit on any ref; unreferenced by any tracked file.

EXPECTED. Left in place, untracked. Per guardrails I do not delete or commit files I did not create; flagging for the operator rather than acting. It affects no gate, no DoD item, and no verdict.

Blocked

(none)