cc-ci/machine-docs/STATUS-redfix.md

STATUS — phase redfix

Phase SSOT: /srv/cc-ci/cc-ci-plan/plan-phase-redfix-canon-sweep-failures.md

Mission: investigate every canon-sweep failure (discourse, mattermost-lts, mumble, bluesky-pds, gitea, keycloak) → isolate → root-cause → classify (flake vs genuine; recipe vs test vs warm-machinery vs load) → FIX each (recipe PR or harness improvement) → verify green. No standing exceptions. Nothing merged.

DONE — 2026-07-09T00:18Z

Phase redfix COMPLETE. All six canon-sweep failures investigated in isolation, root-caused, classified, FIXED — each via a recipe PR or a harness improvement — and verified green; no recipe left as a standing exception; nothing merged (operator merges). Every gate has a fresh Adversary PASS in REVIEW-redfix.md and there is no standing VETO:

• M1 PASS @ 2026-06-18T01:18Z (investigation/classification cold-verified).
• M2 PASS @ 2026-07-09T00:18Z (F-redfix-4 remedy cold-verified; VETO CLEARED; supersedes the 2026-06-18T07:06Z M2 PASS, which was given against parent 07fc6d4).

Adversary findings F-redfix-1/2/3/4 are all CLOSED; no open blocking finding.

The four A-redfix-* findings are also non-blocking and are accounted for here, so nothing dangles:

• A-redfix-1 (/etc/cc-ci/.git/config is 0644 and embeds the Gitea bot password) — OPEN, out of redfix scope, pre-existing undeclared server state, no DoD item. Mirrors my B-redfix-7. Not created by this phase. It is the SAME credential as B-redfix-8, not merely a similar one — verified first-hand 2026-07-09T09:1xZ: the password extracted from the .git/config remote URL and the GITEA_PASSWORD in the orchestrator's /srv/cc-ci/.testenv both hash to 3fcea78925015fc9 (neither is the empty-string artifact e3b0c44298fc1c14). Two exposures, one credential, still unrotated. Folded into the B-redfix-8 remedy below, because rotating the password does not scrub this file.
• A-redfix-2 (the B-redfix-5 wedge's self-heal mechanism) — PARTIALLY WITHDRAWN by the Adversary at wake #44. Its first half stands (a code path does redeploy keycloak — the fresh-deploy branch). Its second half ("no re-trigger; heals on reboot") is false and is superseded by A-redfix-4.
• A-redfix-3 (the main.go "both clones" evidence) — ACCEPTED; that evidence is retracted below. /srv/cc-ci is a symlink to /srv/cc-ci-orch. The file's origin remains unexplained; risk inert.
• A-redfix-4 (a weekly nightly-sweep.timer does re-trigger reconcile; the wedge is silent and recurring) — ACCEPTED and independently re-verified against the node, not taken on authority. The B-redfix-5 precondition below is rewritten accordingly: two unfalsifiable probes deleted, the wedge described as a weekly alternating DOWN/UP outage that raises no alert, and the precondition re-anchored from "slot clean at merge time" to "never deploy 07fc6d4". The trap is not armed and cannot be armed by autonomous node activity (origin/main, the tree the sweep runs, has WARM_CANONICAL = False).

All four corrected operator-facing descriptions, not DoD items. No VETO at wake #43 (b358b7e) or wake #44 (84a1666, 2026-07-09T08:47Z). Note on provenance: A-redfix-2's false half entered this file at f64d102 because I adopted the Adversary's stated mechanism and its two probe commands without re-deriving them. The probes returned the all-clear for a reason unrelated to the property under test (they grep the app name; the timer is named nightly-sweep). Both are now deleted, and every claim in the rewritten block below was re-checked first-hand.

⚠ OPERATOR ACTION REQUIRED — B-redfix-8 (HIGH, open, outside DoD)

Phase DoD is met, but one HIGH item is open and only an operator can close it. It does not block ## DONE (no DoD item depends on it) and is recorded in full in BACKLOG-redfix.md.

WHAT. The live Gitea bot password (GITEA_PASSWORD in /srv/cc-ci/.testenv) was committed in 14c7dee (machine-docs/BACKLOG-redfix.md), pushed to origin/main, and is served in cleartext to the unauthenticated public internet by the public mirror. It is push-capable to recipe-maintainers/*. HEAD and the redaction commit e99e2b3 are clean; only the historical commit 14c7dee serves it. Redaction cannot unpublish it — the value is permanent in history and in every clone.

WHERE. git.autonomic.zone/recipe-maintainers/cc-ci → /raw/commit/14c7dee/machine-docs/BACKLOG-redfix.md

HOW to confirm exposure (unauthenticated fetch; a raw blob at a fixed sha is immutable, so served size must equal object size):

git cat-file -s 14c7dee:machine-docs/BACKLOG-redfix.md    # → 33408

An unauthenticated fetch of the raw URL returning HTTP 200 / 33408 bytes is the leak, not an error page.

HOW to confirm it is still the live credential (commits to the value without republishing it). Run this on the orchestrator — /srv/cc-ci/.testenv lives there, not on cc-ci:

python3 -c 'import hashlib,re;v=re.search(r"^GITEA_PASSWORD=(.*)$",open("/srv/cc-ci/.testenv").read(),re.M).group(1).strip().strip(chr(34)).strip(chr(39));print(hashlib.sha256(v.encode()).hexdigest()[:16])'

EXPECTED. Prints 3fcea78925015fc9 while still unrotated. A different digest ⇒ rotated ⇒ 14c7dee is inert ⇒ close B-redfix-8 — but only if the probe was sound. Two known-bad probes yield a different digest without any rotation, and must NOT be used to close this item:

• e3b0c44298fc1c14 is the empty-input tell, not a rotation. Running the command over ssh cc-ci fails to open .testenv, hashes the empty string, and prints sha256("")[:16] = e3b0c44298fc1c14 (check: printf '' | sha256sum). Fix the probe; do not close B-redfix-8.
• Do not conflate the three digests. 3fcea78925015fc9 = sha256(credential value)[:16] — the only rotation sentinel. 1994fd8d… = sha256(the whole served blob) — immutable, changes never. e3b0c44298fc1c14 = sha256(empty) — a broken probe.

Exposure confirmed at value level (Adversary, 2026-07-09T07:34Z): the currently-valid, push-capable password appears verbatim in the publicly served body, not merely "a blob that once held a secret."

REMEDY (operator). Rotate the Gitea bot password, then update both places that hold it. The credential is a Class-A1 external infra input (plan §4.4) — the Builder must not rotate or invent it. History rewrite is not available to the Builder either (--force is forbidden by the standing rules), so rotation is the only action that actually revokes the exposure.

1. Rotate the bot password in Gitea.

2. Update GITEA_PASSWORD in /srv/cc-ci/.testenv on the orchestrator.

3. Rewrite /etc/cc-ci/.git/config on cc-ci (A-redfix-1) — rotation does NOT scrub it. That file is 644 root:root and embeds the same password in the origin URL's userinfo. /etc/cc-ci is a manual git clone, not nix-generated (nix/hosts/cc-ci-hetzner/configuration.nix:7 documents the clone), so no nixos-rebuild regenerates or scrubs it. Strip the userinfo from the URL rather than re-embedding the new password — re-embedding recreates the exposure verbatim:

   ssh cc-ci 'git -C /etc/cc-ci remote set-url origin \
     https://git.autonomic.zone/recipe-maintainers/cc-ci.git && chmod 600 /etc/cc-ci/.git/config'
   

   Stripping the userinfo does not break the clone: the sweep only ever fetches, and the mirror serves this repo anonymously (that is precisely what B-redfix-8 exploits). Verified 2026-07-09T09:1xZ — GIT_TERMINAL_PROMPT=0 git -c credential.helper= ls-remote https://git.autonomic.zone/recipe-maintainers/cc-ci.git HEAD → f0372b8… HEAD, rc=0, no credential. The sibling cc-ci-secrets submodule URL (.git/config:15) already carries no userinfo, so this matches the existing pattern.

   Independently cold-verified by the Adversary (REVIEW-redfix.md wake #46, 3f27cc6), which attempted to refute it three ways and failed on all three — so this step is safe to run on the node:

   • No push through origin. The only git push in the deployed tree (scripts/recipe-mirror-sync.sh:84-85) remote adds a gitea OAuth-token remote inside a recipe clone; it never touches /etc/cc-ci's origin.
   • The secrets submodule survives the strip. Its URL already carries no userinfo and cc-ci-secrets.git is anonymously fetchable (ls-remote rc=0), so submodule update still works.
   • Nothing auto-pulls the checkout. /etc/cc-ci is clone-once + local nixos-rebuild switch; the sweep never git pulls it. (/root/.git-credentials exists but no credential.helper is wired at global or system scope, so it is inert — the anonymous ls-remote results above are genuinely credential-free.)

Rotation blast radius is small — no operational reason to delay. GITEA_PASSWORD is consumed by exactly one script, scripts/bootstrap-drone-oauth.sh (git grep -ln GITEA_PASSWORD -- . returns that script, docs/ install.md, and machine-docs prose; nothing else in the runner). In particular scripts/recipe-mirror-sync.sh pushes with an OAuth token read from /run/secrets/bridge_gitea_token (:30), not this password, so mirror sync is unaffected by the rotation.

HOW to confirm BOTH exposures are closed (each prints a digest; a value other than 3fcea78925015fc9 on both, and not the empty-string tell e3b0c44298fc1c14, means rotated):

# exposure 2 — the embedded copy; `python3` is absent on cc-ci, so use sha256sum
ssh cc-ci 'sed -n "s#^\s*url = https://[^:]*:\([^@]*\)@git\.autonomic\.zone/recipe-maintainers/cc-ci\.git#\1#p" \
  /etc/cc-ci/.git/config | head -1 | tr -d "\n" | sha256sum | cut -c1-16'

EXPECTED after step 3: the command prints nothing (no userinfo left to match) — an empty result here is the fix, whereas an e3b0c44298fc1c14 from the step-2 probe is a broken probe. Do not confuse them.

Merge target: redfix-m2-harness @ b5f2b10 (not 07fc6d4 — that tip carries the F-redfix-4 defect). The earlier ## DONE of 2026-06-18T07:09Z was withdrawn on 2026-07-09 under the standing VETO F-redfix-4 and is re-asserted here only after the remedy was Adversary-verified. Details of that cycle, and the still-open non-blocking B-redfix-5, are below.

One deferred, non-blocking item remains recorded (NOT part of any DoD item, NOT a standing finding): B-redfix-5 in BACKLOG-redfix.md — in warm_reconcile.py, TWO post-abra.undeploy() calls sit outside the upgrade's try/except: warmsnap.snapshot() on the upgrade path (:514) and warmsnap.restore() on the rollback path (:536). If either raises, live keycloak is left undeployed, and because both sit upstream of every write_alert() call site the failure is silent. Pre-dates the enrollment (present at 07fc6d4). Adversary concurred it is not a VETO. Its consequence is worse than first recorded — see the merge precondition below: reached only via a foreign live-slot meta, which nothing on the node can create today, but if ever reached it produces a weekly alternating SSO outage that reports Result=success.

MERGE PRECONDITION for b5f2b10. The F-redfix-4 fix adds a raise path to both of those calls — warmsnap._assert_slot_not_foreign() (warmsnap.py:158 in snapshot(), :209 in restore()). It raises iff the live slot's snapshot/meta.json records a domain other than the one passed. Composed with B-redfix-5, a foreign live-slot meta wedges live keycloak: on the upgrade path this fires on every stateful auto-upgrade, not only on rollback. It is unreachable on cc-ci today because the live slot holds no snapshot at all (read_meta → None → "a slot with no snapshot yet is free to claim"). It is unreachable for that reason — not because F-redfix-4 is closed.

THE BINDING RULE: never deploy 07fc6d4. (Re-anchored 2026-07-09 per Adversary A-redfix-4; the earlier "verify the slot is clean at merge time" framing named the wrong landmark — a git merge executes nothing.) The only state that creates a foreign live-slot meta is a canonical keycloak seed executed from a tree that carries the enrollment but not canonical_ns(). Exactly one such tree exists: 07fc6d4 (tests/keycloak/recipe_meta.py → WARM_CANONICAL = True, no canonical_ns()). b5f2b10 ships enrollment and fix together, so its canonical seeds land in canon-keycloak/ and the live slot never goes foreign. The precondition is therefore self-maintaining across the merge — deploying the merge target cannot arm the trap; only deploying the intermediate 07fc6d4 can.

No autonomous node activity can arm it. The tree the weekly sweep executes is CCCI_REPO=/etc/cc-ci = origin/main, where WARM_CANONICAL = False. Only 07fc6d4 and b5f2b10 have True.

If it is ever armed, the wedge is SILENT and RECURRING — a weekly alternating outage of the shared SSO provider. (Corrected 2026-07-09 per A-redfix-4, which withdrew the "no re-trigger / heals on reboot" half of A-redfix-2. My own earlier "recovery is manual" (68b51d5) was also wrong. Re-derived independently against the node — see HOW below.) warm_reconcile.py has two invokers, and the second is a weekly timer:

nightly-sweep.timer  (OnCalendar=Sun *-*-* 03:00:00, Persistent=true)
  └─ nightly_sweep.main():147 → roll_warm_infra() → subprocess warm_reconcile.py keycloak
     (WARM_APPS = ["keycloak", "traefik"], nightly_sweep.py:39,57-61)

Once armed, with current != latest: sweep N → abra.undeploy() → snapshot() guard raises → keycloak DOWN. Sweep N+1 (7 days later) → is_deployed is False → the fresh-deploy branch (:471-479), which never calls warmsnap → keycloak UP on the old version. Sweep N+2 → upgrade path → DOWN again. Alternating weeks, indefinitely, because the foreign snapshot/meta.json is never removed. (abra.undeploy() (abra.py:295-297) runs only abra app undeploy -n and does not remove the app .env, so current_version stays resolvable — this is why the fresh-deploy branch is the one taken.)

And it raises no alert. roll_warm_infra() captures the subprocess rc and discards it (nightly_sweep.py:59-62 — rc is only ever printed), so nightly-sweep.service still reports Result=success. The raise at :514 is outside the only try/except (:520-525, which wraps deploy_version + wait_healthy — that gap is B-redfix-5) and upstream of every write_alert() call site (:493,500,503,539). reconcile()'s sole caller main():556 does not catch either. A weekly SSO outage would surface nowhere. (All line numbers here are b5f2b10 coordinates, matching the rest of this block; on main the same sites sit two lines earlier.)

REMEDIATION if ever armed: delete the foreign snapshot/ from the slot (or fix B-redfix-5). Redeploying keycloak does NOT fix it — the next sweep re-wedges. This is the one part of the old text that was right, and it is the part that matters operationally.

HOW to verify (from a clone at b5f2b10). Note the probe greps the callee, never the app name: the invoking timer is called nightly-sweep, so list-timers | grep -i keycloak and git grep warm-keycloak -- nix/ are both empty while a timer drives it weekly. Those two probes were in this file until 2026-07-09 and are unfalsifiable by construction; they have been deleted, not fixed.

git grep -n warm_reconcile -- runner/ nix/            # SOUND probe: finds both invokers
git show redfix-m2-harness:runner/harness/abra.py     | sed -n '295,297p'   # undeploy: no .env removal
git show redfix-m2-harness:runner/warm_reconcile.py   | sed -n '471,479p'   # fresh-deploy branch, no warmsnap
ssh cc-ci 'systemctl cat nightly-sweep.timer | grep -iE "OnCalendar|Persistent"'
ssh cc-ci 'systemctl list-timers --all nightly-sweep.timer --no-pager'

EXPECTED: git grep warm_reconcile prints both nix/modules/warm-keycloak.nix:26 (the oneshot unit) and runner/nightly_sweep.py:60 (the weekly sweep's subprocess); undeploy() is a single _run(["app","undeploy",domain,"-n"], …) with no .env handling; :472 is if not deployed: returning deployed-fresh:{target} at :479 with no warmsnap between; the timer prints OnCalendar=Sun *-*-* 03:00:00 + Persistent=true.

Observational proof the sweep — not the unit — drives reconcile (independent of reading any code; observed 2026-07-09):

ssh cc-ci 'systemctl show warm-keycloak.service -p ExecMainStartTimestamp
           stat -c "%y %n" /var/lib/ci-warm/keycloak/last_good
           systemctl list-timers --all nightly-sweep.timer --no-pager'

EXPECTED: unit last ran Wed 2026-06-17 17:29:31 UTC; last_good mtime 2026-07-05 03:04:51.209; timer last trigger Sun 2026-07-05 03:04:50 UTC. The slot was written 18 days after the unit last ran and 1.2 s after the timer fired, and write_last_good() is reachable only from reconcile() (call sites :478,484,491,527, all inside it) ⇒ reconcile("keycloak") executed from the sweep.

Also: warm-keycloak.service's ExecStart resolves to /nix/store/…-runner/warm_reconcile.py, and harness/warmsnap.py lives in that same derivation. So nixos-rebuild switch re-runs reconcile iff runner/** changed — an unrelated switch heals nothing, while the merge's own activation switch is a reconcile trigger (alongside the Sunday sweep).

Blast radius: keycloak only. warm.WARM_DOMAINS == {"keycloak"} and keycloak is the only SPECS entry with stateful: True (traefik is stateful: False — version-rollback-only, never snapshots). No other warm unit reaches either call site.

HOW to verify the trap is not armed (on cc-ci; python3 is absent there, so ls/cat):

ssh cc-ci 'ls -A /var/lib/ci-warm/keycloak/; cat /var/lib/ci-warm/keycloak/snapshot/meta.json 2>&1'

EXPECTED (observed 2026-07-09T08:5xZ): last_good only, and cat reports No such file or directory — no snapshot/, hence no meta.json; no /var/lib/ci-warm/canon-keycloak either; live keycloak 200. A meta.json whose "domain" is anything other than warm-keycloak.ci.commoninternet.net means the trap is armed: delete that foreign snapshot/ (or fix B-redfix-5) before the next Sunday sweep, not merely before the merge. A meta.json recording warm-keycloak.ci.commoninternet.net is self-consistent and passes the guard.

---

F-redfix-4 remedy (the reason DONE was withdrawn and re-asserted) — 2026-07-09

WHAT. F-redfix-4 is fixed at redfix-m2-harness @ b5f2b10 (parent 07fc6d4, the sha the M2 PASS was given against). Warm state is now keyed by a stack namespace, not a bare recipe: canonical.canonical_ns(r) is the single source from which BOTH the canonical's domain and its warm-state slot derive. A live-warm provider (r in warm.WARM_DOMAINS) gets ns canon-<r> → domain warm-canon-keycloak.ci.commoninternet.net (unchanged) + slot /var/lib/ci-warm/canon-keycloak/. Every other recipe keeps ns <r> (the invariant is structural — r not in warm.WARM_DOMAINS — not a property of any particular count). WARM_DOMAINS == {"keycloak"} is a singleton, so the re-key is provably keycloak -> canon-keycloak and nothing else: of the 21 enrolled recipes the other 20 satisfy canonical_ns(r) == r, so zero existing canonical slots change. On live disk that is 17 seeded canonicals (those carrying a canonical.json), all of which remain in prune_stale's keep set.

Addresses all four consequences: (1)+(2) slots disjoint, neither deployment can replace the other's known-good; (3) the outage race is removed with the shared slot; (4) prune_stale's reconciler-dir invariant is now structural — <recipe>/ never gains a canonical.json.

Also: warmsnap._assert_slot_not_foreign() refuses to snapshot/restore a slot recorded against a different domain (defence in depth; fails before the destructive swap, not at the next restore). The two comments asserting the deployments "can never touch each other" are corrected (canonical.py, tests/keycloak/recipe_meta.py). WARM_CANONICAL = True is retained — keycloak stays enrolled, no skip-guard, no silent de-enrollment.

MIGRATION: none required. On cc-ci /var/lib/ci-warm/keycloak/ contains only last_good (the canonical was never seeded), so no existing file changes slot. Verify: ls -A /var/lib/ci-warm/keycloak/ → last_good only.

WHERE. cc-ci repo, branch redfix-m2-harness @ b5f2b10. Files: runner/harness/warmsnap.py, runner/harness/canonical.py, runner/warm_reconcile.py, runner/run_recipe_ci.py, tests/keycloak/recipe_meta.py, tests/unit/test_warmsnap.py, tests/unit/test_canonical.py.

HOW to verify (1) — unit suite, cold clone, no docker needed.

git clone -q --branch redfix-m2-harness <cc-ci remote> /tmp/v && cd /tmp/v
/nix/store/x188l04r3gfkh18gy1dpf05fv3kkrgs7-python3-3.12.8-env/bin/python3 -m pytest tests/unit -q

EXPECTED: 325 passed (baseline at parent 07fc6d4 is 315 passed; +10 F-redfix-4 regression tests). The 10 include test_live_and_canonical_slots_are_disjoint, test_slot_maps_1to1_to_its_stack, test_registry_path_of_live_warm_provider_is_not_the_reconciler_dir, test_prune_stale_deenrolled_provider_spares_reconciler_last_good, test_snapshot_refuses_to_clobber_another_domains_slot, test_restore_refuses_a_foreign_slot.

HOW to verify (2) — the VETO's clearing condition, on the real node. Non-destructive: writes only to a scratch CCCI_WARM_ROOT + one throwaway docker volume. Uses the real idle warm-canon-keycloak stack for the canonical side and a throwaway warm-fakelive_…_data volume as the live-warm stand-in (the live warm-keycloak stack cannot be undeployed to snapshot it).

ssh cc-ci
docker volume create warm-fakelive_ci_commoninternet_net_data
MP=$(docker volume inspect -f '{{.Mountpoint}}' warm-fakelive_ci_commoninternet_net_data); echo x > $MP/live.txt
git clone -q --branch redfix-m2-harness <cc-ci remote> /tmp/v && mkdir -p /tmp/vw/keycloak
echo '10.7.1+26.6.2' > /tmp/vw/keycloak/last_good
cd /tmp/v/runner && CCCI_WARM_ROOT=/tmp/vw /nix/store/jag2131a95gw6ng7grig9pj3dn2q8vrv-python3-3.12.8-env/bin/python3 - <<'PY'
import sys; sys.path.insert(0, ".")
from harness import warmsnap as ws, canonical as c
LIVE, CANON = ws.live_slot("keycloak"), c.canonical_slot("keycloak")
CANON_DOM, LIVE_DOM = c.canonical_domain("keycloak"), "warm-fakelive.ci.commoninternet.net"
print(ws.snap_dir(LIVE), ws.snap_dir(CANON), c.registry_path("keycloak"), sep="\n")
ws.snapshot(CANON, CANON_DOM, version="canon-known-good")
ws.snapshot(LIVE,  LIVE_DOM,  version="live-last-good")     # used to clobber the canonical
print("restore(canon) ->", ws.restore(CANON, CANON_DOM)["volumes"])
print("restore(live)  ->", ws.restore(LIVE,  LIVE_DOM )["volumes"])
print("last_good:", open("/tmp/vw/keycloak/last_good").read().strip())
try: ws.snapshot(CANON, LIVE_DOM); print("FAIL: foreign snapshot allowed")
except ws.SnapshotError as e: print("foreign REFUSED:", str(e)[:60])
PY
docker volume rm warm-fakelive_ci_commoninternet_net_data; rm -rf /tmp/vw /tmp/v

EXPECTED (observed 2026-07-09):

/tmp/vw/keycloak/snapshot                     <- live slot
/tmp/vw/canon-keycloak/snapshot               <- canonical slot: DISJOINT
/tmp/vw/canon-keycloak/canonical.json         <- registry NOT in the reconciler's dir
restore(canon) -> ['warm-canon-keycloak_ci_commoninternet_net_mariadb',
                   'warm-canon-keycloak_ci_commoninternet_net_providers']
restore(live)  -> ['warm-fakelive_ci_commoninternet_net_data']
last_good: 10.7.1+26.6.2                      <- survived the canonical seed
foreign REFUSED: warm slot 'canon-keycloak' holds the known-good of ...

Each restore() returns its OWN stack's volumes (the clearing condition). Pre-fix, the same script prints one shared slot /tmp/vw/keycloak/snapshot for both and restore(canon) raises SnapshotError.

Node integrity after my own run of the above (2026-07-09). Real warm root untouched (/var/lib/ci-warm/keycloak/ = last_good only); warm-canon-keycloak mariadb digest byte-identical across the restore round-trip (1201440268 48846 before and after, 391 files / 2 files); live warm-keycloak…/realms/master → 200 throughout; throwaway volume removed; scratch removed.

Scope. keycloak only. M1 classifications and the discourse / mattermost-lts / gitea / bluesky-pds / mumble fixes are untouched by this commit and remain Adversary-verified (M1 PASS @2026-06-18T01:18Z; the five non-keycloak M2 fixes content-verified through re-confirmation #7).

Known residual, NOT in F-redfix-4's clearing condition (filed as B-redfix-5 in BACKLOG-redfix.md, not blocking): warm_reconcile.py's rollback warmsnap.restore() still sits outside the try/except that guards the upgrade, so any restore failure (e.g. a corrupt/absent snapshot) leaves live keycloak undeployed after abra.undeploy(). The F-redfix-4 race that made this reachable is gone; the pre-existing robustness gap is not. Recorded, not silently dropped.

The prior DONE text is retained verbatim below as the historical record of what was claimed on 2026-06-18.

---

(HISTORICAL — withdrawn 2026-07-09 under VETO F-redfix-4; superseded by the DONE above) DONE — 2026-06-18T07:09Z

Phase redfix COMPLETE. All six canon-sweep failures investigated in isolation, root-caused, classified, FIXED — each via a recipe PR or a harness improvement — and verified green; no recipe left as a standing exception; nothing merged (operator merges). Both gates have a fresh Adversary PASS in REVIEW-redfix.md with no standing VETO:

• M1 PASS @ 2026-06-18T01:18Z (investigation/classification cold-verified).
• M2 PASS @ 2026-06-18T07:06Z (all 6 fixes cold-verified; supersedes the 06:42Z FAIL after the discourse F-redfix-1 rework).

Fixes (per recipe): mattermost-lts recipe PR #1 (pg_backup.sh + restore.post-hook) — restore round-trips; discourse recipe PR #4 @9ff5e19 (official-image migration + drop orphaned sidekiq from compose.smtpauth.yml) — level=5, lint R011 ✅; keycloak harness (collision-free warm-canon-<r> + enroll) — promotes without touching live SSO; mumble harness (handshake budget 60→180s) — flake stabilized, non-weakening; gitea recipe PR #2 @a0f2db8 (app.ini seed-on-empty into writable volume) — M1 read-only crash gone; bluesky-pds recipe PR #4 @4987ba9 (caddy ${STACK_NAME}_app) — warm health 200 (was 000). gitea/bluesky end-to-end canonical advance is operator-merge-gated (fix proven by chaos-deploy; published tags don't carry it pre-merge) — consistent with "nothing merged", not a shrug.

Evidence addendum 2026-07-08 (re: F-redfix-3) — discourse sha pins have rotted; verify by CONTENT

The discourse shas cited below (9ff5e19, 53ba0910) no longer resolve on the mirror. A later phase force-pushed and extended the shared branch discourse-official-image (now ede6399 = refs/pull/5/head; redfix's PR is no longer #4). This is branch drift by later work, not a retraction of the redfix fix, and it does not disturb the M2 PASS — which was given against the shas that existed on 2026-06-18. The other three recipe pins and the harness pin are exact and still resolve: mattermost-lts ci/pg-restore@4ca7f418, gitea ci/app-ini-writable@a0f2db88, bluesky-pds ci/warm-routing-alias@4987ba91, cc-ci redfix-m2-harness@07fc6d4a.

Historical sha lines below are left unedited on purpose — they record what was verified when. Use this durable content assertion instead of the shas to re-verify discourse at any future head:

git clone https://git.autonomic.zone/recipe-maintainers/discourse && cd discourse
git show origin/discourse-official-image:compose.yml | grep -m1 'image:.*discourse'
git show origin/discourse-official-image:compose.smtpauth.yml | grep -c sidekiq

EXPECTED: image: discourse/discourse:3.5.3 (the official-image migration = M2's claim) and 0 (the F-redfix-1 orphaned-sidekiq removal). Both re-confirmed by the Builder at ede6399 on 2026-07-08, and by the Adversary at ede6399 and refs/pull/4/head@0c4539b7. (Caveat when reproducing absence of a sha: a --filter=blob:none clone and git fetch origin <sha> both yield false "absent" signals — test reachability from all refs/heads/* + refs/pull/*/head instead.)

---

Phase: M1 — investigate + isolate + classify (IN PROGRESS)

Bootstrapped 2026-06-17T23:20Z. cc-ci healthy, no run in flight, next scheduled sweep 2026-06-21 (3-day clear window). Disk / 38G free (75% used).

Isolation harness (how I reproduce each failure ALONE)

Each canon-sweep per-recipe run is runner/nightly_sweep.run_on_tag(recipe, latest): abra.recipe_checkout(recipe, <latest-tag>) then run_recipe_ci.py with RECIPE=<r> CCCI_SKIP_FETCH=1 and REF/QUICK/MODE/VERSION unset (cold, full, head==tag). Isolation = run ONE recipe at a time with NO concurrent sweep load on the single node (the loaded node is the known flake source per phase plan §2.1). Runs execute on cc-ci from /etc/cc-ci.

Starting canonical state (cc-ci /var/lib/ci-warm/<r>/canonical.json, read 2026-06-17T23:19Z)

Recipe	Canonical now	Note
discourse	(none)	no canonical dir
mattermost-lts	(none)	no canonical dir
mumble	1.0.0+v1.6.870-0 @ 20260617T180501Z	canonical PRESENT, written TODAY — flake signal
bluesky-pds	(none)	no canonical dir
gitea	3.5.3+1.24.2-rootless @ 20260617T083930Z	3.6.0 advance not promoted
keycloak	(none)	de-enrolled (WARM_CANONICAL off)

M1 investigation tracker

Recipe	Isolation run	Result	Root cause	Classification
discourse	DONE @23:40Z (/tmp/redfix-discourse.log on cc-ci)	install/backup/restore/custom PASS; upgrade overlay FAIL. Deploys+serves fine — NOT a timeout/FATA.	cc-ci overlay tests/discourse/test_upgrade.py asserts head runs official discourse/discourse:3.5.3 + drops sidekiq; latest tag 0.8.1+3.5.0 AND main both still bitnamilegacy/discourse:3.5.0+sidekiq (migration exists in no release/main). The depends_on discourse string is a non-fatal prepull-only warning, not the deploy.	stale/PR-specific cc-ci OVERLAY test mismatched to canonical-sweep context (not flake/timeout/recipe-deploy/warm-machinery)
mattermost-lts	DONE @00:05Z (/tmp/redfix-mattermost-lts.log)	install/upgrade/backup/custom PASS; restore FAIL ci_marker does not exist — deterministic in isolation (not a load race)	recipe postgres svc backup labels: backs up hot live PGDATA + dump but has NO backupbot.restore.post-hook to replay the dump → restore doesn't round-trip postgres. Contrast immich (passes): dump-only backup.volumes.postgres.path: backup.sql + restore.post-hook: /pg_backup.sh restore.	genuine RECIPE defect at latest → recipe PR (adopt immich-style dump+restore-post-hook)
mumble	DONE — 2× isolation GREEN (/tmp/redfix-mumble.log + /tmp/redfix-mumble2.log)	ALL tiers PASS incl. handshake on BOTH runs; no orphans; canonical re-promoted green each time	handshake (TLS+ServerSync) not completing within ~60s retry under heavy concurrent sweep load; fine in isolation	load/timing FLAKE → harness stabilization (readiness gate / retry)
bluesky-pds	DONE @00:45Z (/tmp/redfix-bluesky-pds.log + live diag)	cold lifecycle GREEN; WC5 promote 000 reproduces (warm /xrpc/_health last status 0). NOT a flake	caddy on-demand TLS (ask http://app:3000/tls-check) can't reach app: caddy resolves bare app to OTHER stacks' app endpoints on shared proxy net (getent app→only 10.10.0.X, never internal 10.0.3.3; proxy has drone/traefik/keycloak/ccci app aliases) → no cert → 000. Promote machinery correct (refused to write canonical).	genuine routing/RECIPE defect (cross-stack app-alias collision on shared proxy) → recipe PR: unique PDS service name/alias. NOT promote-machinery, NOT flake
gitea	DONE @00:14Z (/tmp/redfix-gitea2.log + live container logs)	cold lifecycle (incl fresh 3.5.3→3.6.0 upgrade) PASS; warm advance crash-loops	LoadCommonSettings() [F] error saving JWT Secret … failed to save "/etc/gitea/app.ini": read-only file system — gitea 3.6.0/1.24.2 tries to persist a JWT to the read-only app.ini docker-config mount on warm reattach (before DB migration; 3.5.3 data intact). Cold passes (fresh secrets, no rewrite).	genuine RECIPE defect (3.6.0 + read-only app.ini config mount on advance) → recipe PR: render app.ini into the writable config volume. (1st gitea run hit a nixenv "already deployed" leftover confound — fixed by undeploying to idle then re-running)
keycloak	DONE @01:05Z (code-verified; no run)	de-enrolled. canonical_domain("keycloak") == WARM_DOMAINS["keycloak"] == warm-keycloak.ci.commoninternet.net EXACTLY (canonical.py:42, warm.py:27,44). Live keycloak 200 /realms/master.	data-warm canonical domain uses same warm-<r> scheme as the live-warm OIDC provider → promote would collide with live shared SSO. No collision-free canonical namespace exists.	HARNESS defect (warm-domain namespace collision) → fix: collision-free canonical_domain for live-warm providers (warm-canon-<r>), then enroll keycloak

M1 results table (recipe → failure → isolation result → root cause → classification → fix approach)

Recipe	Canon-sweep failure	Isolation result	Flake or genuine	Root cause	Class	Fix approach (M2)
discourse	"cold-deploy timeout / deploy FATA"	install/backup/restore/custom GREEN; upgrade overlay RED	genuine (deterministic) — but the canon root-cause was WRONG (no timeout, no deploy FATA)	cc-ci overlay tests/discourse/test_upgrade.py asserts head = official discourse/discourse:3.5.3 + sidekiq dropped; that migration is in NO release tag and NOT in main (all use bitnamilegacy/discourse:3.5.0+sidekiq)	stale/PR-specific cc-ci OVERLAY test	make the overlay assert migration-faithfulness only when the head IS that migration (not vs a release tag), OR a recipe PR migrating off deprecated bitnamilegacy — settle in M2 (NOT a test-weakening)
mattermost-lts	test_restore_returns_state RED	install/upgrade/backup/custom GREEN; restore RED (ci_marker does not exist)	genuine (deterministic in isolation) — NOT the canon "loaded-node race"	recipe postgres backup labels back up hot PGDATA + a dump but have no backupbot.restore.post-hook to replay it; restore doesn't round-trip. immich (passes) uses dump-only path + restore.post-hook	genuine RECIPE defect	recipe PR: adopt immich-style postgres dump + backupbot.restore.post-hook replay
mumble	test_handshake… RED	ALL tiers GREEN in isolation (×N) incl. handshake	FLAKE (load/timing)	handshake (TLS+ServerSync) doesn't complete within the 60s retry under heavy concurrent sweep load; fine isolated; canonical written green today	load/concurrency FLAKE	harness stabilization: stronger readiness gate before the custom tier / longer-or-smarter handshake retry
bluesky-pds	warm promote /xrpc/_health → 000	cold lifecycle GREEN; warm promote 000 reproduces	genuine (deterministic) — NOT a load/rate-limit flake	caddy on-demand TLS calls http://app:3000/tls-check; caddy resolves bare app to OTHER stacks' app endpoints on the shared proxy net (every stack aliases its main svc app), never bluesky's own internal app (10.0.3.3) → connection refused → no cert → 000	genuine ROUTING/RECIPE defect (cross-stack app-alias collision)	recipe PR: give the PDS service a unique name/alias so caddy resolves only bluesky's app
gitea	3.5.3→3.6.0 warm advance doesn't promote	cold (incl fresh upgrade) GREEN; warm advance crash-loops	genuine (deterministic)	gitea 3.6.0/1.24.2 saves a JWT secret to /etc/gitea/app.ini on warm reattach; app.ini is a read-only docker-config mount → read-only file system FATA at LoadCommonSettings (pre-migration; 3.5.3 data intact). Cold passes (fresh secrets, no rewrite)	genuine RECIPE defect	recipe PR: render app.ini into the writable config:/etc/gitea volume (entrypoint) instead of a read-only docker config
keycloak	de-enrolled (not tested)	code-verified (no run)	genuine (structural)	canonical_domain("keycloak") == WARM_DOMAINS["keycloak"] == warm-keycloak.ci.commoninternet.net EXACTLY → a data-warm canonical would collide with the live-warm OIDC provider	HARNESS defect (warm-domain namespace collision)	harness: collision-free canonical_domain for live-warm providers (warm-canon-<r>), then enroll keycloak (WARM_CANONICAL=True)

HOW the Adversary cold-verifies each classification (run ONE recipe at a time, no concurrent load)

Isolation invocation (per recipe R at latest tag T), from /etc/cc-ci on cc-ci: git -C ~/.abra/recipes/R checkout -f --quiet T && env -u REF -u CCCI_QUICK -u MODE -u VERSION RECIPE=R CCCI_SKIP_FETCH=1 cc-ci-run runner/run_recipe_ci.py Latest tags: discourse 0.8.1+3.5.0, mattermost-lts 2.1.9+10.11.15, mumble 1.0.0+v1.6.870-0, bluesky-pds 0.3.0+v0.4.219, gitea 3.6.0+1.24.2-rootless.

• discourse — EXPECT install/backup/restore/custom pass, upgrade fail on test_head_runs_official_image_not_bitnamilegacy + test_sidekiq_service_dropped_by_head. Confirm the overlay mismatch statically: git -C ~/.abra/recipes/discourse show 0.8.1+3.5.0:compose.yml | grep -A1 ' app:' and ... show main:compose.yml both = bitnamilegacy/discourse:3.5.0; grep -c 'sidekiq:' = 1 in both. So the test's discourse/discourse:3.5.3/no-sidekiq expectation exists nowhere upstream.
• mattermost-lts — EXPECT restore fail relation "ci_marker" does not exist. Confirm root cause statically: git -C ~/.abra/recipes/mattermost-lts show 2.1.9+10.11.15:compose.yml | grep backupbot shows pre-hook + backup.path but NO restore.post-hook; immich git -C ~/.abra/recipes/immich show <latest>:compose.yml | grep backupbot shows restore.post-hook: /pg_backup.sh restore.
• mumble — EXPECT all tiers green (run 2–3× to confirm reproducibly green isolated). Canonical written green: cat /var/lib/ci-warm/mumble/canonical.json.
• bluesky-pds — EXPECT cold green, WC5 promote !! WC5 promote failed … warm-bluesky-pds … last status 0. While the warm stack is up, confirm root cause: caddy logs dial tcp 10.10.0.X:3000: connect: connection refused for app:3000/tls-check; docker exec <caddy> getent hosts app returns proxy IPs (10.10.0.X), the app's real internal IP is 10.0.3.x; docker network inspect proxy | grep _app shows many stacks aliasing app. (Tear down the orphaned warm-bluesky-pds stack + volumes after.)
• gitea — REQUIRES idle canonical first: if warm-gitea is deployed, docker stack rm warm-gitea_ci_commoninternet_net (retains data+config volumes) so the advance reattaches from idle. EXPECT cold green, warm advance crash-loop with container log LoadCommonSettings() [F] error saving JWT Secret … "/etc/gitea/app.ini": read-only file system. Restore: leave warm-gitea undeployed (idle 3.5.3, volumes retained) — registry stays 3.5.3+1.24.2-rootless.
• keycloak — no run. Code-verify: canonical.canonical_domain('keycloak') → warm.stable_domain('keycloak') → warm-keycloak.ci.commoninternet.net; warm.WARM_DOMAINS['keycloak'] == same string (runner/harness/canonical.py:42-44, warm.py:27-29,44-48). Live keycloak 200 on /realms/master.

Node state left clean

All isolation runs torn down; orphaned warm-bluesky-pds stack+volumes removed; warm-gitea restored to idle 3.5.3 (volumes retained, registry unchanged); only live warm-keycloak deployed (healthy). No run_recipe_ci.py processes.

M1 — PASS @ 2026-06-18T01:18Z (REVIEW-redfix.md; all 6 classifications cold-verified CORRECT by Adversary's own isolation re-runs). No VETO. Cleared to M2.

Phase: M2 — FIX + verify all six (IN PROGRESS)

Fix designs locked in BACKLOG-redfix.md. Recipe PRs (mattermost-lts/bluesky/gitea) on git.autonomic.zone mirrors via the recipe mirror+PR flow, verified !testme (NEVER merge). Harness fixes (keycloak/mumble) on a cc-ci branch, verified via the harness. discourse: overlay-scope decision. Node now free for my deploys (Adversary done with M1).

M2 fix tracker (updated 2026-06-18T05:53Z — ALL VERIFIED)

Recipe	Class	Fix	PR/branch + ref	Status
mattermost-lts	recipe defect	pg_backup.sh + backupbot.restore.post-hook (immich pattern)	mirror PR #1 ci/pg-restore @4ca7f418	VERIFIED — !testme run #901 ALL tiers green incl test_restore_returns_state
discourse	stale cc-ci overlay	recipe: bitnamilegacy->official discourse image migration + drop orphaned image-less sidekiq from compose.smtpauth.yml (F-redfix-1)	mirror PR #4 discourse-official-image @9ff5e19	VERIFIED — own cold run /tmp/redfix-discourse-m2verify.log level=5 of 5 (all tiers + lint R011 PASS); F-redfix-1 regression fixed
keycloak	harness defect	collision-free canonical_domain (warm-canon-<r> for WARM_DOMAINS recipes) + enroll	cc-ci branch redfix-m2-harness @61211db	VERIFIED — branch-checkout run promotes at warm-canon-keycloak; live warm-keycloak 200 throughout
mumble	load/timing flake	harness: handshake readiness budget 60s->180s	cc-ci branch redfix-m2-harness @07fc6d4	VERIFIED — branch-checkout run all tiers green incl handshake; budget active+non-regressing
gitea	recipe defect	app.ini->staging /etc/gitea/app.ini.init + docker-setup seed-on-EMPTY + DOCKER_SETUP_SH_VERSION v3	mirror PR #2 ci/app-ini-writable @a0f2db8	VERIFIED (direct chaos-deploy; promote merge-gated — see below)
bluesky-pds	recipe defect (routing)	caddy {$APP_HOST}=${STACK_NAME}_app (operator: NO rename) + CADDYFILE_VERSION v2	mirror PR #4 ci/warm-routing-alias @4987ba9	VERIFIED (direct chaos-deploy; promote merge-gated — see below)

cc-ci-side change verification: run from a checkout of redfix-m2-harness (CCCI_REPO=); never touches /etc/cc-ci main. redfix-m2-harness is now mumble+keycloak ONLY (bluesky needs no cc-ci change with the ${STACK_NAME}_app approach; the rename's exec-ref commit b96b8a4 was dropped).

Gate: M2 — RE-CLAIMED, awaiting Adversary (2026-06-18T06:55Z; orig claim 05:53Z)

Re-claim delta (addresses Adversary M2 FAIL @06:42Z — finding F-redfix-1). The first M2 verdict was FAIL on discourse ONLY (other 5 PASS, do-not-redo). F-redfix-1: the official-image migration dropped sidekiq from compose.yml but left a dangling image-less sidekiq: block in compose.smtpauth.yml → L5 lint R011 fail (run level=4) + broken SMTP-auth deploy. FIXED in PR #4 discourse-official-image @9ff5e19 (force-pushed onto @53ba0910): dropped the orphaned sidekiq: block; the app: override already carries DISCOURSE_SMTP_PASSWORD_FILE + smtp_password secret (sidekiq is internal to the official image), so no SMTP coverage lost. grep sidekiq compose*.yml = 0. VERIFIED two ways: (1) the Adversary's exact lint.py repro flow at 9ff5e19 → R011 ✅; (2) my own full cold run /tmp/redfix-discourse-m2verify.log → RUN SUMMARY ... level=5 of 5, all tiers pass (install/upgrade/backup/restore/custom), lint rung: pass. Node clean: no discourse stack, NO discourse canonical (untagged migrated head correctly does not promote — should_promote tagged-gate), recipe reset to published tag 0.8.1+3.5.0. The other 5 fixes are unchanged since their Adversary PASS (keycloak, mumble, gitea, bluesky-pds, mattermost-lts) — no re-run needed.

Adversary cold-verify for discourse: clone discourse @9ff5e19, run RECIPE=discourse CCCI_SKIP_FETCH=1 … run_recipe_ci.py → EXPECT level=5 of 5 (lint R011 ✅, all tiers pass, both upgrade-overlay tests test_head_runs_official_image_not_bitnamilegacy + test_sidekiq_service_dropped_by_head pass); OR the lint-only repro in F-redfix-1 → R011 ✅. grep -c sidekiq ~/.abra/recipes/discourse/compose*.yml @9ff5e19 = 0.

---

Gate: M2 — original claim (2026-06-18T05:53Z)

WHAT (M2 DoD). All six canon-sweep failures FIXED — each via a recipe PR or a harness improvement — and verified green. No recipe left as a standing exception. Nothing merged (operator merges). Per recipe:

• mattermost-lts (recipe PR #1) — added pg_backup.sh + postgres backupbot.restore.post-hook so the logical dump round-trips on restore.
• discourse (recipe PR #4) — migrated the head off deprecated bitnamilegacy to the official discourse/discourse image so the stale PR-faithfulness overlay (test_head_runs_official_image…, test_sidekiq_service_dropped…) passes on the migrated head (NOT a test-weakening).
• keycloak (harness branch) — canonical_domain returns a collision-free warm-canon-<r> for recipes in warm.WARM_DOMAINS (live-warm OIDC providers); keycloak enrolled (WARM_CANONICAL=True).
• mumble (harness branch) — handshake readiness budget widened 60s->180s (load-flake stabilization).
• gitea (recipe PR #2) — app.ini is now seeded into the WRITABLE /etc/gitea volume by docker-setup (if [ ! -s /etc/gitea/app.ini ], seed-on-EMPTY) from the read-only staging config app.ini.init; DOCKER_SETUP_SH_VERSION v1->v3 forces the new docker-setup to re-mount. Gitea 1.24.2 can then persist its JWT secret (the M1 read-only-app.ini crash is gone).
• bluesky-pds (recipe PR #4) — caddy resolves its OWN app via the fully-qualified swarm name ${STACK_NAME}_app (caddy {$APP_HOST} env, set in the caddy service) instead of bare app, which collided with other stacks' app aliases on the shared proxy net. CADDYFILE_VERSION v1->v2.

HOW + EXPECTED + WHERE (Adversary cold-verify, one recipe at a time, no concurrent load):

• mattermost-lts — read-only artifact: /var/lib/cc-ci-runs/901/ on cc-ci — all tiers pass, junit/restore__cc-ci__test_restore.xml testsuite failures=0, test_restore_returns_state pass. OR re-run !testme on PR #1 @4ca7f418. EXPECT restore green.
• discourse — !testme on PR #4 @53ba0910 (run #849 green) OR run from a checkout of the migrated head: EXPECT install/backup/restore/custom + upgrade overlay all pass (head now official image).
• keycloak — from a redfix-m2-harness @61211db checkout (CCCI_REPO=), run RECIPE=keycloak CCCI_SKIP_FETCH=1 ... run_recipe_ci.py. EXPECT all cold tiers pass + WC5 promote succeeds at domain warm-canon-keycloak.ci.commoninternet.net (NOT warm-keycloak); live warm-keycloak.ci.commoninternet.net/realms/master stays 200 throughout. Code: canonical.py canonical_domain returns warm-canon- for r in warm.WARM_DOMAINS.
• mumble — from redfix-m2-harness @07fc6d4 checkout, run RECIPE=mumble CCCI_SKIP_FETCH=1 …. EXPECT all 5 tiers green incl custom/test_protocol_handshake.py::test_handshake_completes_with_ channel_presence; handshake budget = 36 attempts / 180s (was 60s). (Load-flake is not deterministically reproducible; this verifies the stabilization is applied, sound, non-weakening.)
• gitea (recipe PR #2 @a0f2db8 on mirror branch ci/app-ini-writable) — DIRECT chaos-deploy proof (the harness WC5 promote is merge-gated, see NOTE). With the idle 3.5.3 canonical present: cd ~/.abra/recipes/gitea && git checkout -f a0f2db8 then chaos-deploy onto the retained canonical volumes (0-byte app.ini = genuine pre-fix 3.5.3 state): abra app deploy warm-gitea.ci.commoninternet.net -C -o -n. EXPECT: service 1/1; the config volume's app.ini seeded 0->~1862 bytes (INSTALL_LOCK = true); /api/v1/version -> 200 {"version":"1.24.2"} and /api/healthz -> 200 (curl inside the app container); retained 3.5.3 data adopted (data dirs dated 2026-06-17T08:39); ZERO read-only file system crashes in docker service logs (M1 crashed here). Evidence: /tmp/redfix-gitea-m2-directproof.log on cc-ci. Teardown: abra app undeploy … -n, truncate the volume app.ini to 0 (restore pre-fix state). canonical.json stays 3.5.3 idle e6a1cc79.
• bluesky-pds (recipe PR #4 @4987ba9 on mirror branch ci/warm-routing-alias) — DIRECT chaos-deploy proof (warm-promote is the only failing path; merge-gated). git checkout -f 4987ba9; generate secrets (abra app secret generate warm-bluesky-pds.ci.commoninternet.net --all -m -C -o -n) + insert a PLC rotation key (tests/bluesky-pds/install_steps.sh logic: 32-byte hex into pds_plc_rotation_key v1); re-checkout 4987ba9 AFTER secret ops (abra secret insert force-fetches+reverts the checkout); abra app deploy warm-bluesky-pds.ci.commoninternet.net -C -o -n (EXPECT caddyfile: v1 -> v2, NEW DEPLOYMENT 4987ba9). EXPECT: app+caddy 1/1; inside caddy getent hosts warm-bluesky-pds_ci_commoninternet_net_app -> a 10.0.x.x INTERNAL ip (own stack) while getent hosts app -> a 10.10.x.x proxy ip (foreign, the M1 collision); caddy log "certificate obtained successfully" with 0 "connection refused"; external curl https://warm-bluesky-pds.ci. commoninternet.net/xrpc/_health -> 200 {"version":"0.4.219"} (M1 was 000). Evidence: /tmp/redfix-bluesky-m2-directproof.log. Teardown: undeploy + remove volumes (caddy_data, pds_data)
  • secrets (no canonical, matching M1).

NOTE — gitea & bluesky end-to-end canonical-promote is OPERATOR-MERGE-GATED (not a shrug). The harness WC5 promote does a recipe_checkout(published-tag)+non-chaos deploy, and BOTH run_recipe_ci.py:373 AND abra force-fetch refs/tags/* from upstream (abra.py:135 documents this), so any local move of the release tag to the fix commit is reverted to the PUBLISHED commit. The published 3.6.0 / 0.3.0 tags do NOT yet carry the fix (PR not merged — operator merges, per phase guardrail), so pre-merge the promote necessarily deploys the unfixed published release. Confirmed empirically: a full gitea harness run's WC5 promote deployed 357926f and crash-looped exactly like M1. The DIRECT chaos-deploy (chaos = deploy the working-tree checkout = the PR fix) is therefore the MAXIMAL + faithful pre-merge proof — it reproduces the EXACT M1 failing scenario (gitea: the retained canonical volumes; bluesky: warm-bluesky-pds on the shared proxy) and shows the fix resolves it. End-to-end canonical advance follows automatically once the operator merges PR #2 / #4 and the release tag carries the fix. This is NOT a standing exception — the defect is fixed + proven; only the registry-advance awaits the operator's merge (the phase's own "nothing merged" constraint).

WHERE (refs). Recipe PRs on git.autonomic.zone/recipe-maintainers/<recipe>: mattermost-lts ci/pg-restore@4ca7f418, discourse discourse-official-image@53ba0910, gitea ci/app-ini-writable @a0f2db8, bluesky-pds ci/warm-routing-alias@4987ba9. cc-ci harness branch redfix-m2-harness@07fc6d4 (keycloak 61211db + mumble 07fc6d4). Reasoning/dead-ends in JOURNAL-redfix.md. Node left clean (only infra + live warm-keycloak 200; gitea idle 3.5.3 volumes retained, canonical e6a1cc79 unchanged; no bluesky/test stacks/volumes/secrets; no run procs).

Gate: M1 — PASS (above).

WHAT (M1 DoD). All six canon-sweep failures investigated in ISOLATION (one recipe at a time, no concurrent sweep load), root-caused with first-hand evidence, and classified (flake vs genuine; recipe vs test vs warm-machinery vs load) — see the M1 results table + HOW the Adversary cold-verifies sections above. Summary: discourse = stale cc-ci overlay test (canon timeout/FATA root-cause was wrong); mattermost-lts = genuine recipe defect (no backupbot.restore.post-hook); mumble = load/timing FLAKE (2× isolation green); bluesky-pds = genuine routing defect (caddy↔app app-alias collision on shared proxy); gitea = genuine recipe defect (read-only app.ini config mount + 3.6.0 JWT save); keycloak = harness warm-domain namespace collision. NO "probably a flake" — every classification has an isolation re-run or code proof.

HOW + EXPECTED + WHERE. Per-recipe cold-verify commands, expected outputs, and evidence paths are in the two sections above ("M1 results table" and "HOW the Adversary cold-verifies each classification"). Evidence logs on cc-ci: /tmp/redfix-{discourse,mattermost-lts,mumble,mumble2,bluesky-pds,gitea2}.log. Reasoning/dead-ends in JOURNAL-redfix.md. Node left clean (see "Node state left clean" above).

Untracked file main.go — NOT produced by this phase (operator: unexplained)

WHAT. An untracked main.go (281 bytes, a Go "Hello, World!" net/http server on :8080) sits in the repo root. It is not in any commit on any ref, is not gitignored, and is unrelated to every redfix DoD item. No go.mod accompanies it. I did not create it and have not committed, modified, or deleted it.

WHERE. /srv/cc-ci-orch/cc-ci/main.go — one file in one clone.

RETRACTED (2026-07-09, Adversary A-redfix-3): an earlier version of this entry claimed the file was "present in both clones … i.e. written by something outside git." That evidence is void. /srv/cc-ci is a symlink to /srv/cc-ci-orch, so /srv/cc-ci/cc-ci/main.go and /srv/cc-ci-orch/cc-ci/main.go are the same inode — one file seen twice, not two clones. The "a git operation cannot leave the same untracked file in two clones" inference therefore carries no information about the file's origin, which remains unexplained. Independently re-derived: ls -la /srv/ shows cc-ci -> /srv/cc-ci-orch, and stat -c '%i %h %n' on both paths prints the same inode 3254604 with links=1.

HOW to verify. git log --all --oneline -- main.go → empty (in no commit). git status --short → ?? main.go. ls -la /srv/ | grep cc-ci → cc-ci -> /srv/cc-ci-orch (the symlink that voids the "both clones" claim). stat -c '%i %h %n' /srv/cc-ci/cc-ci/main.go /srv/cc-ci-orch/cc-ci/main.go → same inode, links=1.

Risk: inert (Adversary-confirmed). 281-byte hello-world net/http listener; no go.mod; go is not installed; nothing listening on :8080; in no commit on any ref; unreferenced by any tracked file.

EXPECTED. Left in place, untracked. Per guardrails I do not delete or commit files I did not create; flagging for the operator rather than acting. It affects no gate, no DoD item, and no verdict.

Blocked

(none)