cc-ci/machine-docs/STATUS-redfix.md

# STATUS — phase `redfix`

Phase SSOT: `/srv/cc-ci/cc-ci-plan/plan-phase-redfix-canon-sweep-failures.md`

Mission: investigate every canon-sweep failure (discourse, mattermost-lts, mumble, bluesky-pds,
gitea, keycloak) → isolate → root-cause → classify (flake vs genuine; recipe vs test vs
warm-machinery vs load) → FIX each (recipe PR or harness improvement) → verify green. No standing
exceptions. Nothing merged.

## DONE — 2026-07-09T00:18Z

Phase `redfix` COMPLETE. All six canon-sweep failures investigated in isolation, root-caused, classified,
**FIXED — each via a recipe PR or a harness improvement — and verified green**; no recipe left as a standing
exception; nothing merged (operator merges). Every gate has a fresh Adversary PASS in REVIEW-redfix.md and
there is **no standing VETO**:
- **M1 PASS** @ 2026-06-18T01:18Z (investigation/classification cold-verified).
- **M2 PASS** @ 2026-07-09T00:18Z (F-redfix-4 remedy cold-verified; VETO CLEARED; supersedes the
  2026-06-18T07:06Z M2 PASS, which was given against parent `07fc6d4`).

Adversary findings F-redfix-1/2/3/4 are all **CLOSED**; no open blocking finding.

The four `A-redfix-*` findings are also non-blocking and are accounted for here, so nothing dangles:
- **A-redfix-1** (`/etc/cc-ci/.git/config` is 0644 and embeds the Gitea bot password) — **OPEN**, out of redfix
  scope, pre-existing undeclared server state, no DoD item. Mirrors my B-redfix-7. Not created by this phase.
  **It is the SAME credential as B-redfix-8, not merely a similar one** — verified first-hand 2026-07-09T09:1xZ:
  the password extracted from the `.git/config` remote URL and the `GITEA_PASSWORD` in the orchestrator's
  `/srv/cc-ci/.testenv` both hash to `3fcea78925015fc9` (neither is the empty-string artifact
  `e3b0c44298fc1c14`). **Two exposures, one credential, still unrotated.** Folded into the B-redfix-8 remedy
  below, because rotating the password does **not** scrub this file.
- **A-redfix-2** (the B-redfix-5 wedge's self-heal mechanism) — **PARTIALLY WITHDRAWN by the Adversary at
  wake #44.** Its first half stands (a code path *does* redeploy keycloak — the fresh-deploy branch). Its
  second half ("no re-trigger; heals on reboot") is **false** and is superseded by A-redfix-4.
- **A-redfix-3** (the `main.go` "both clones" evidence) — **ACCEPTED**; that evidence is retracted below.
  `/srv/cc-ci` is a symlink to `/srv/cc-ci-orch`. The file's origin remains unexplained; risk inert.
- **A-redfix-4** (a weekly `nightly-sweep.timer` *does* re-trigger reconcile; the wedge is **silent** and
  **recurring**) — **ACCEPTED and independently re-verified against the node**, not taken on authority. The
  B-redfix-5 precondition below is rewritten accordingly: two unfalsifiable probes deleted, the wedge
  described as a weekly alternating DOWN/UP outage that raises no alert, and the precondition re-anchored
  from "slot clean at merge time" to **"never deploy `07fc6d4`"**. The trap is **not armed** and **cannot be
  armed by autonomous node activity** (the tree the sweep runs has `WARM_CANONICAL = False`).

  **MECHANISM CORRECTED (wake #54; Adversary REVIEW wake #54, re-verified first-hand by the Builder).** The
  sweep does **not** run `origin/main`. `nightly-sweep.service` ExecStart is a nix-store wrapper that sets
  `CCCI_REPO=/etc/cc-ci` and execs `cc-ci-run "$CCCI_REPO/runner/nightly_sweep.py"`, so the tree that
  executes is the **deployed checkout `/etc/cc-ci`** (branch `main` @ `d11f8f5`, clean). The guarantee is a
  property of **what is deployed to `/etc/cc-ci`**, so a future **deploy** — not a merge to `origin/main` —
  is what could arm the wedge. Conclusion unchanged (trap disarmed); only the named tree was wrong. Verify:

      ssh cc-ci 'git -C /etc/cc-ci rev-parse --abbrev-ref HEAD; grep -n "^WARM_CANONICAL" /etc/cc-ci/tests/keycloak/recipe_meta.py;
        git -C /etc/cc-ci merge-base --is-ancestor 07fc6d4 HEAD && echo ARMED || echo disarmed'
      # EXPECTED: main / 16:WARM_CANONICAL = False / disarmed   (a real assignment, not the line-15 comment)

All four corrected operator-facing *descriptions*, not DoD items. No VETO at wake #43 (`b358b7e`) or wake #44
(`84a1666`, 2026-07-09T08:47Z). **Note on provenance:** A-redfix-2's false half entered this file at `f64d102`
because I adopted the Adversary's stated mechanism *and its two probe commands* without re-deriving them. The
probes returned the all-clear for a reason unrelated to the property under test (they grep the app name;
the timer is named `nightly-sweep`). Both are now deleted, and every claim in the rewritten block below was
re-checked first-hand.

## ⚠ OPERATOR ACTION REQUIRED — B-redfix-8 (HIGH, open, outside DoD)

Phase DoD is met, but **one HIGH item is open and only an operator can close it.** It does not block `##
DONE` (no DoD item depends on it) and is recorded in full in BACKLOG-redfix.md.

**WHAT.** The live Gitea bot password (`GITEA_PASSWORD` in `/srv/cc-ci/.testenv`) was committed in
`14c7dee` (`machine-docs/BACKLOG-redfix.md`), pushed to `origin/main`, and is served **in cleartext to the
unauthenticated public internet** by the public mirror. It is push-capable to `recipe-maintainers/*`. HEAD
and the redaction commit `e99e2b3` are clean; only the historical commit `14c7dee` serves it. Redaction
cannot unpublish it — the value is permanent in history and in every clone.

**WHERE.** `git.autonomic.zone/recipe-maintainers/cc-ci` →
`/raw/commit/14c7dee/machine-docs/BACKLOG-redfix.md`

**HOW to confirm exposure** (unauthenticated fetch; a raw blob at a fixed sha is immutable, so served size
must equal object size):

    git cat-file -s 14c7dee:machine-docs/BACKLOG-redfix.md    # → 33408

An unauthenticated fetch of the raw URL returning **HTTP 200 / 33408 bytes** is the leak, not an error page.

**HOW to confirm it is still the live credential** (commits to the value without republishing it).
Run this **on the orchestrator** — `/srv/cc-ci/.testenv` lives there, **not** on `cc-ci`:

    python3 -c 'import hashlib,re;v=re.search(r"^GITEA_PASSWORD=(.*)$",open("/srv/cc-ci/.testenv").read(),re.M).group(1).strip().strip(chr(34)).strip(chr(39));print(hashlib.sha256(v.encode()).hexdigest()[:16])'

**EXPECTED.** Prints `3fcea78925015fc9` while still unrotated. **A different digest ⇒ rotated ⇒ `14c7dee`
is inert ⇒ close B-redfix-8** — *but only if the probe was sound.* Two known-bad probes yield a different
digest without any rotation, and must NOT be used to close this item:

- **`e3b0c44298fc1c14` is the empty-input tell, not a rotation.** Running the command over `ssh cc-ci`
  fails to open `.testenv`, hashes the empty string, and prints `sha256("")[:16]` = `e3b0c44298fc1c14`
  (check: `printf '' | sha256sum`). Fix the probe; do not close B-redfix-8.
- **Do not conflate the three digests.** `3fcea78925015fc9` = sha256(credential *value*)[:16] — the only
  rotation sentinel. `1994fd8d…` = sha256(the whole served *blob*) — immutable, changes never.
  `e3b0c44298fc1c14` = sha256(empty) — a broken probe.

Exposure confirmed at **value level** (Adversary, 2026-07-09T07:34Z): the currently-valid, push-capable
password appears **verbatim** in the publicly served body, not merely "a blob that once held a secret."

**REMEDY (operator).** Rotate the Gitea bot password, then update **both** places that hold it. The credential
is a Class-A1 external infra input (plan §4.4) — the Builder must not rotate or invent it. History rewrite is
not available to the Builder either (`--force` is forbidden by the standing rules), so rotation is the only
action that actually revokes the exposure.

1. Rotate the bot password in Gitea.
2. Update `GITEA_PASSWORD` in `/srv/cc-ci/.testenv` **on the orchestrator**.
3. **Rewrite `/etc/cc-ci/.git/config` on `cc-ci` (A-redfix-1) — rotation does NOT scrub it.** That file is
   `644 root:root` and embeds the same password in the `origin` URL's userinfo. `/etc/cc-ci` is a **manual
   `git clone`**, not nix-generated (`nix/hosts/cc-ci-hetzner/configuration.nix:7` documents the clone), so no
   `nixos-rebuild` regenerates or scrubs it. Strip the userinfo from the URL rather than re-embedding the new
   password — re-embedding recreates the exposure verbatim:

       ssh cc-ci 'git -C /etc/cc-ci remote set-url origin \
         https://git.autonomic.zone/recipe-maintainers/cc-ci.git && chmod 600 /etc/cc-ci/.git/config'

   **CORRECTION (wake #54, Builder, 2026-07-09T10:2xZ): `/etc/cc-ci/.git/config` is 1 of 78 world-readable
   copies — step 3 alone fixes 1/78.** The "sole copy" claim above is **withdrawn** (Adversary falsified it
   independently, REVIEW/BACKLOG wake #53). Scope, re-derived first-hand — 78 world-readable cred-bearing
   `.git/config`, all carrying sentinel `3fcea78925015fc9` (= the same live password):
   **68** under `/var/lib/cc-ci-runs/manual-*/abra/recipes/*/`, **8** in `/nix/store` (0444, read-only, clears
   on GC), **1** `/tmp/v`, **1** `/etc/cc-ci`. Verify:

       ssh cc-ci 'find / -xdev -name config -path "*/.git/*" 2>/dev/null | while read f; do
         grep -q "autonomic-bot:" "$f" || continue
         setpriv --reuid=1000 --regid=1000 --clear-groups cat "$f" >/dev/null 2>&1 && echo "$f"; done | wc -l'
       # EXPECTED: 78   (after remediation: 0, ignoring /nix/store)

4. **Scrub the other 77 (A-redfix-1, widened).** The generator is the **credentialed `origin` in the canonical
   clones** `/root/.abra/recipes/*/.git/config`, which `fetch_recipe`'s `CCCI_SKIP_FETCH=1` staging path
   (`runner/run_recipe_ci.py:348-353`, `shutil.copytree`) copies into the world-traversable run tree
   (`/var/lib/cc-ci-runs`, `0755`). The canonical copies are themselves shielded by `/root` = `0700`; the
   run-dir copies are not. Strip userinfo at the source, then scrub the copies:

       ssh cc-ci 'for d in /root/.abra/recipes/*/; do r=$(basename "$d");
         git -C "$d" remote set-url origin https://git.autonomic.zone/recipe-maintainers/$r.git 2>/dev/null; done
       rm -rf /tmp/v
       find /var/lib/cc-ci-runs -path "*/.git/config" -exec grep -lq "autonomic-bot:" {} \; -delete'

   Stripping the userinfo does **not** break the clone: the sweep only ever *fetches*, and the mirror serves
   this repo anonymously (that is precisely what B-redfix-8 exploits). Verified 2026-07-09T09:1xZ —
   `GIT_TERMINAL_PROMPT=0 git -c credential.helper= ls-remote https://git.autonomic.zone/recipe-maintainers/cc-ci.git HEAD`
   → `f0372b8… HEAD`, rc=0, no credential. The sibling `cc-ci-secrets` submodule URL (`.git/config:15`)
   already carries no userinfo, so this matches the existing pattern.

   **Independently cold-verified by the Adversary (REVIEW-redfix.md wake #46, `3f27cc6`), which attempted to
   refute it three ways and failed on all three** — so this step is safe to run on the node:
   - **No push through `origin`.** The only `git push` in the deployed tree (`scripts/recipe-mirror-sync.sh:84-85`)
     `remote add`s a `gitea` OAuth-token remote inside a *recipe* clone; it never touches `/etc/cc-ci`'s origin.
   - **The `secrets` submodule survives the strip.** Its URL already carries no userinfo and
     `cc-ci-secrets.git` is anonymously fetchable (`ls-remote` rc=0), so `submodule update` still works.
   - **Nothing auto-pulls the checkout.** `/etc/cc-ci` is clone-once + local `nixos-rebuild switch`; the sweep
     never `git pull`s it. (`/root/.git-credentials` exists but no `credential.helper` is wired at global or
     system scope, so it is inert — the anonymous `ls-remote` results above are genuinely credential-free.)

**Rotation blast radius is small — no operational reason to delay.** `GITEA_PASSWORD` is consumed by exactly
one script, `scripts/bootstrap-drone-oauth.sh` (`git grep -ln GITEA_PASSWORD -- .` returns that script, `docs/
install.md`, and machine-docs prose; nothing else in the runner). In particular `scripts/recipe-mirror-sync.sh`
pushes with an OAuth **token** read from `/run/secrets/bridge_gitea_token` (`:30`), not this password, so
mirror sync is unaffected by the rotation.

**HOW to confirm BOTH exposures are closed** (each prints a digest; a value other than `3fcea78925015fc9` on
both, and not the empty-string tell `e3b0c44298fc1c14`, means rotated):

    # exposure 2 — the embedded copy; `python3` is absent on cc-ci, so use sha256sum
    ssh cc-ci 'sed -n "s#^\s*url = https://[^:]*:\([^@]*\)@git\.autonomic\.zone/recipe-maintainers/cc-ci\.git#\1#p" \
      /etc/cc-ci/.git/config | head -1 | tr -d "\n" | sha256sum | cut -c1-16'

**EXPECTED after step 3:** the command prints **nothing** (no userinfo left to match) — an empty result here
is the fix, whereas an `e3b0c44298fc1c14` from the *step-2* probe is a broken probe. Do not confuse them.

**Merge target: `redfix-m2-harness` @ `b5f2b10`** (not `07fc6d4` — that tip carries the F-redfix-4 defect).
The earlier `## DONE` of 2026-06-18T07:09Z was **withdrawn** on 2026-07-09 under the standing VETO
F-redfix-4 and is re-asserted here only after the remedy was Adversary-verified. Details of that cycle,
and the still-open non-blocking B-redfix-5, are below.

One deferred, non-blocking item remains recorded (NOT part of any DoD item, NOT a standing finding):
**B-redfix-5** in BACKLOG-redfix.md — in `warm_reconcile.py`, TWO post-`abra.undeploy()` calls sit outside
the upgrade's `try/except`: `warmsnap.snapshot()` on the upgrade path (`:514`) and `warmsnap.restore()` on
the rollback path (`:536`). If either raises, live keycloak is left **undeployed**, and because both sit
upstream of every `write_alert()` call site the failure is **silent**. Pre-dates the enrollment (present at
`07fc6d4`). Adversary concurred it is not a VETO. Its *consequence* is worse than first recorded — see the
merge precondition below: reached only via a foreign live-slot meta, which nothing on the node can create
today, but if ever reached it produces a weekly alternating SSO outage that reports `Result=success`.

**MERGE PRECONDITION for `b5f2b10`.** The F-redfix-4 fix *adds* a raise path to both of those calls —
`warmsnap._assert_slot_not_foreign()` (`warmsnap.py:158` in `snapshot()`, `:209` in `restore()`). It raises
iff the live slot's `snapshot/meta.json` records a `domain` other than the one passed. Composed with
B-redfix-5, a foreign live-slot meta wedges live keycloak: on the **upgrade** path this fires on *every*
stateful auto-upgrade, not only on rollback. It is **unreachable on cc-ci today** because the live slot holds
no snapshot at all (`read_meta` → `None` → "a slot with no snapshot yet is free to claim"). It is unreachable
for that reason — **not** because F-redfix-4 is closed.

**THE BINDING RULE: never deploy `07fc6d4`.** (Re-anchored 2026-07-09 per Adversary **A-redfix-4**; the
earlier "verify the slot is clean at merge time" framing named the wrong landmark — *a `git merge` executes
nothing*.) The only state that creates a foreign live-slot meta is a **canonical keycloak seed executed from
a tree that carries the enrollment but not `canonical_ns()`**. Exactly one such tree exists: `07fc6d4`
(`tests/keycloak/recipe_meta.py` → `WARM_CANONICAL = True`, no `canonical_ns()`). `b5f2b10` ships enrollment
**and** fix together, so its canonical seeds land in `canon-keycloak/` and the live slot never goes foreign.
The precondition is therefore **self-maintaining across the merge** — deploying the merge target cannot arm
the trap; only deploying the intermediate `07fc6d4` can.

**No autonomous node activity can arm it.** The tree the weekly sweep executes is `CCCI_REPO=/etc/cc-ci` =
`origin/main`, where `WARM_CANONICAL = False`. Only `07fc6d4` and `b5f2b10` have `True`.

**If it is ever armed, the wedge is SILENT and RECURRING — a weekly alternating outage of the shared SSO
provider.** (Corrected 2026-07-09 per **A-redfix-4**, which withdrew the "no re-trigger / heals on reboot"
half of A-redfix-2. My own earlier "recovery is manual" (`68b51d5`) was also wrong. Re-derived independently
against the node — see HOW below.) `warm_reconcile.py` has **two** invokers, and the second is a weekly timer:

    nightly-sweep.timer  (OnCalendar=Sun *-*-* 03:00:00, Persistent=true)
      └─ nightly_sweep.main():147 → roll_warm_infra() → subprocess warm_reconcile.py keycloak
         (WARM_APPS = ["keycloak", "traefik"], nightly_sweep.py:39,57-61)

Once armed, with `current != latest`: **sweep N** → `abra.undeploy()` → `snapshot()` guard raises → keycloak
**DOWN**. **Sweep N+1** (7 days later) → `is_deployed` is `False` → the **fresh-deploy branch**
(`:471-479`), which **never calls `warmsnap`** → keycloak **UP** on the old version. **Sweep N+2** → upgrade
path → **DOWN** again. **Alternating weeks, indefinitely**, because the foreign `snapshot/meta.json` is never
removed. (`abra.undeploy()` (`abra.py:295-297`) runs only `abra app undeploy -n` and does **not** remove the
app `.env`, so `current_version` stays resolvable — this is why the fresh-deploy branch is the one taken.)

**And it raises no alert.** `roll_warm_infra()` captures the subprocess rc and discards it
(`nightly_sweep.py:59-62` — `rc` is only ever printed), so `nightly-sweep.service` still reports
`Result=success`. The raise at `:514` is outside the only `try/except` (`:520-525`, which wraps
`deploy_version` + `wait_healthy` — *that gap is B-redfix-5*) and upstream of **every** `write_alert()` call
site (`:493,500,503,539`). `reconcile()`'s sole caller `main():556` does not catch either. A weekly SSO
outage would surface **nowhere**. (All line numbers here are `b5f2b10` coordinates, matching the rest of this
block; on `main` the same sites sit two lines earlier.)

**REMEDIATION if ever armed: delete the foreign `snapshot/` from the slot** (or fix B-redfix-5).
**Redeploying keycloak does NOT fix it** — the next sweep re-wedges. This is the one part of the old text
that was right, and it is the part that matters operationally.

**HOW to verify** (from a clone at `b5f2b10`). Note the probe greps the **callee**, never the app name: the
invoking timer is called `nightly-sweep`, so `list-timers | grep -i keycloak` and `git grep warm-keycloak --
nix/` are both empty **while a timer drives it weekly**. Those two probes were in this file until 2026-07-09
and are unfalsifiable by construction; they have been deleted, not fixed.

    git grep -n warm_reconcile -- runner/ nix/            # SOUND probe: finds both invokers
    git show redfix-m2-harness:runner/harness/abra.py     | sed -n '295,297p'   # undeploy: no .env removal
    git show redfix-m2-harness:runner/warm_reconcile.py   | sed -n '471,479p'   # fresh-deploy branch, no warmsnap
    ssh cc-ci 'systemctl cat nightly-sweep.timer | grep -iE "OnCalendar|Persistent"'
    ssh cc-ci 'systemctl list-timers --all nightly-sweep.timer --no-pager'

**EXPECTED:** `git grep warm_reconcile` prints both `nix/modules/warm-keycloak.nix:26` (the oneshot unit) and
`runner/nightly_sweep.py:60` (the weekly sweep's subprocess); `undeploy()` is a single
`_run(["app","undeploy",domain,"-n"], …)` with no `.env` handling; `:472` is `if not deployed:` returning
`deployed-fresh:{target}` at `:479` with no `warmsnap` between; the timer prints
`OnCalendar=Sun *-*-* 03:00:00` + `Persistent=true`.

**Observational proof the sweep — not the unit — drives reconcile** (independent of reading any code;
observed 2026-07-09):

    ssh cc-ci 'systemctl show warm-keycloak.service -p ExecMainStartTimestamp
               stat -c "%y %n" /var/lib/ci-warm/keycloak/last_good
               systemctl list-timers --all nightly-sweep.timer --no-pager'

**EXPECTED:** unit last ran `Wed 2026-06-17 17:29:31 UTC`; `last_good` mtime `2026-07-05 03:04:51.209`; timer
last trigger `Sun 2026-07-05 03:04:50 UTC`. The slot was written **18 days after the unit last ran and 1.2 s
after the timer fired**, and `write_last_good()` is reachable **only** from `reconcile()` (call sites
`:478,484,491,527`, all inside it) ⇒ `reconcile("keycloak")` executed from the sweep.

**Also:** `warm-keycloak.service`'s `ExecStart` resolves to `/nix/store/…-runner/warm_reconcile.py`, and
`harness/warmsnap.py` lives in that **same** derivation. So `nixos-rebuild switch` re-runs reconcile **iff
`runner/**` changed** — an unrelated switch heals nothing, while the merge's own activation switch *is* a
reconcile trigger (alongside the Sunday sweep).

**Blast radius: keycloak only.** `warm.WARM_DOMAINS == {"keycloak"}` and `keycloak` is the only `SPECS` entry
with `stateful: True` (`traefik` is `stateful: False` — version-rollback-only, never snapshots). No other
warm unit reaches either call site.

**HOW to verify the trap is not armed** (on cc-ci; `python3` is absent there, so `ls`/`cat`):

    ssh cc-ci 'ls -A /var/lib/ci-warm/keycloak/; cat /var/lib/ci-warm/keycloak/snapshot/meta.json 2>&1'

**EXPECTED (observed 2026-07-09T08:5xZ):** `last_good` only, and `cat` reports `No such file or directory` —
no `snapshot/`, hence no `meta.json`; no `/var/lib/ci-warm/canon-keycloak` either; live keycloak `200`. **A
`meta.json` whose `"domain"` is anything other than `warm-keycloak.ci.commoninternet.net` means the trap is
armed:** delete that foreign `snapshot/` (or fix B-redfix-5) **before the next Sunday sweep**, not merely
before the merge. A `meta.json` recording `warm-keycloak.ci.commoninternet.net` is self-consistent and passes
the guard.

---

### F-redfix-4 remedy (the reason DONE was withdrawn and re-asserted) — 2026-07-09

**WHAT.** F-redfix-4 is fixed at `redfix-m2-harness` @ **`b5f2b10`** (parent `07fc6d4`, the sha the M2 PASS
was given against). Warm state is now keyed by a **stack namespace**, not a bare recipe:
`canonical.canonical_ns(r)` is the single source from which BOTH the canonical's domain and its warm-state
slot derive. A live-warm provider (`r in warm.WARM_DOMAINS`) gets ns `canon-<r>` → domain
`warm-canon-keycloak.ci.commoninternet.net` (**unchanged**) + slot `/var/lib/ci-warm/canon-keycloak/`.
Every other recipe keeps ns `<r>` (the invariant is structural — `r not in warm.WARM_DOMAINS` — not a
property of any particular count). `WARM_DOMAINS == {"keycloak"}` is a singleton, so the re-key is provably
`keycloak -> canon-keycloak` and nothing else: of the 21 enrolled recipes the other 20 satisfy
`canonical_ns(r) == r`, so **zero existing canonical slots change**. On live disk that is 17 seeded
canonicals (those carrying a `canonical.json`), all of which remain in `prune_stale`'s `keep` set.

Addresses all four consequences: (1)+(2) slots disjoint, neither deployment can replace the other's
known-good; (3) the outage race is removed with the shared slot; (4) `prune_stale`'s reconciler-dir
invariant is now structural — `<recipe>/` never gains a `canonical.json`.

Also: `warmsnap._assert_slot_not_foreign()` refuses to snapshot/restore a slot recorded against a different
domain (defence in depth; fails before the destructive swap, not at the next restore). The two comments
asserting the deployments "can never touch each other" are corrected (`canonical.py`,
`tests/keycloak/recipe_meta.py`). `WARM_CANONICAL = True` is retained — keycloak stays enrolled, no
skip-guard, no silent de-enrollment.

**MIGRATION: none required.** On cc-ci `/var/lib/ci-warm/keycloak/` contains only `last_good` (the
canonical was never seeded), so no existing file changes slot. Verify: `ls -A /var/lib/ci-warm/keycloak/`
→ `last_good` only.

**WHERE.** cc-ci repo, branch `redfix-m2-harness` @ `b5f2b10`. Files: `runner/harness/warmsnap.py`,
`runner/harness/canonical.py`, `runner/warm_reconcile.py`, `runner/run_recipe_ci.py`,
`tests/keycloak/recipe_meta.py`, `tests/unit/test_warmsnap.py`, `tests/unit/test_canonical.py`.

**HOW to verify (1) — unit suite, cold clone, no docker needed.**

    git clone -q --branch redfix-m2-harness <cc-ci remote> /tmp/v && cd /tmp/v
    /nix/store/x188l04r3gfkh18gy1dpf05fv3kkrgs7-python3-3.12.8-env/bin/python3 -m pytest tests/unit -q

EXPECTED: `325 passed` (baseline at parent `07fc6d4` is `315 passed`; +10 F-redfix-4 regression tests).
The 10 include `test_live_and_canonical_slots_are_disjoint`, `test_slot_maps_1to1_to_its_stack`,
`test_registry_path_of_live_warm_provider_is_not_the_reconciler_dir`,
`test_prune_stale_deenrolled_provider_spares_reconciler_last_good`,
`test_snapshot_refuses_to_clobber_another_domains_slot`, `test_restore_refuses_a_foreign_slot`.

**HOW to verify (2) — the VETO's clearing condition, on the real node.** Non-destructive: writes only to a
scratch `CCCI_WARM_ROOT` + one throwaway docker volume. Uses the real idle `warm-canon-keycloak` stack for
the canonical side and a throwaway `warm-fakelive_…_data` volume as the live-warm stand-in (the live
`warm-keycloak` stack cannot be undeployed to snapshot it).

    ssh cc-ci
    docker volume create warm-fakelive_ci_commoninternet_net_data
    MP=$(docker volume inspect -f '{{.Mountpoint}}' warm-fakelive_ci_commoninternet_net_data); echo x > $MP/live.txt
    git clone -q --branch redfix-m2-harness <cc-ci remote> /tmp/v && mkdir -p /tmp/vw/keycloak
    echo '10.7.1+26.6.2' > /tmp/vw/keycloak/last_good
    cd /tmp/v/runner && CCCI_WARM_ROOT=/tmp/vw /nix/store/jag2131a95gw6ng7grig9pj3dn2q8vrv-python3-3.12.8-env/bin/python3 - <<'PY'
    import sys; sys.path.insert(0, ".")
    from harness import warmsnap as ws, canonical as c
    LIVE, CANON = ws.live_slot("keycloak"), c.canonical_slot("keycloak")
    CANON_DOM, LIVE_DOM = c.canonical_domain("keycloak"), "warm-fakelive.ci.commoninternet.net"
    print(ws.snap_dir(LIVE), ws.snap_dir(CANON), c.registry_path("keycloak"), sep="\n")
    ws.snapshot(CANON, CANON_DOM, version="canon-known-good")
    ws.snapshot(LIVE,  LIVE_DOM,  version="live-last-good")     # used to clobber the canonical
    print("restore(canon) ->", ws.restore(CANON, CANON_DOM)["volumes"])
    print("restore(live)  ->", ws.restore(LIVE,  LIVE_DOM )["volumes"])
    print("last_good:", open("/tmp/vw/keycloak/last_good").read().strip())
    try: ws.snapshot(CANON, LIVE_DOM); print("FAIL: foreign snapshot allowed")
    except ws.SnapshotError as e: print("foreign REFUSED:", str(e)[:60])
    PY
    docker volume rm warm-fakelive_ci_commoninternet_net_data; rm -rf /tmp/vw /tmp/v

EXPECTED (observed 2026-07-09):

    /tmp/vw/keycloak/snapshot                     <- live slot
    /tmp/vw/canon-keycloak/snapshot               <- canonical slot: DISJOINT
    /tmp/vw/canon-keycloak/canonical.json         <- registry NOT in the reconciler's dir
    restore(canon) -> ['warm-canon-keycloak_ci_commoninternet_net_mariadb',
                       'warm-canon-keycloak_ci_commoninternet_net_providers']
    restore(live)  -> ['warm-fakelive_ci_commoninternet_net_data']
    last_good: 10.7.1+26.6.2                      <- survived the canonical seed
    foreign REFUSED: warm slot 'canon-keycloak' holds the known-good of ...

Each `restore()` returns its OWN stack's volumes (the clearing condition). Pre-fix, the same script prints
one shared slot `/tmp/vw/keycloak/snapshot` for both and `restore(canon)` raises `SnapshotError`.

**Node integrity after my own run of the above (2026-07-09).** Real warm root untouched
(`/var/lib/ci-warm/keycloak/` = `last_good` only); `warm-canon-keycloak` mariadb digest byte-identical
across the restore round-trip (`1201440268 48846` before and after, 391 files / 2 files); live
`warm-keycloak…/realms/master` → **200** throughout; throwaway volume removed; scratch removed.

**Scope.** keycloak only. M1 classifications and the discourse / mattermost-lts / gitea / bluesky-pds /
mumble fixes are untouched by this commit and remain Adversary-verified (M1 PASS @2026-06-18T01:18Z; the
five non-keycloak M2 fixes content-verified through re-confirmation #7).

**Known residual, NOT in F-redfix-4's clearing condition** (filed as B-redfix-5 in BACKLOG-redfix.md, not
blocking): `warm_reconcile.py`'s rollback `warmsnap.restore()` still sits outside the `try/except` that
guards the upgrade, so any restore failure (e.g. a corrupt/absent snapshot) leaves live keycloak undeployed
after `abra.undeploy()`. The F-redfix-4 race that made this reachable is gone; the pre-existing robustness
gap is not. Recorded, not silently dropped.

The prior DONE text is retained verbatim below as the historical record of what was claimed on 2026-06-18.

---

## (HISTORICAL — withdrawn 2026-07-09 under VETO F-redfix-4; superseded by the DONE above) DONE — 2026-06-18T07:09Z

Phase `redfix` COMPLETE. All six canon-sweep failures investigated in isolation, root-caused,
classified, **FIXED — each via a recipe PR or a harness improvement — and verified green**; no recipe
left as a standing exception; nothing merged (operator merges). Both gates have a fresh Adversary PASS
in REVIEW-redfix.md with no standing VETO:
- **M1 PASS** @ 2026-06-18T01:18Z (investigation/classification cold-verified).
- **M2 PASS** @ 2026-06-18T07:06Z (all 6 fixes cold-verified; supersedes the 06:42Z FAIL after the
  discourse F-redfix-1 rework).

Fixes (per recipe): mattermost-lts recipe PR #1 (pg_backup.sh + restore.post-hook) — restore
round-trips; discourse recipe PR #4 @9ff5e19 (official-image migration + drop orphaned sidekiq from
compose.smtpauth.yml) — level=5, lint R011 ✅; keycloak harness (collision-free `warm-canon-<r>` +
enroll) — promotes without touching live SSO; mumble harness (handshake budget 60→180s) — flake
stabilized, non-weakening; gitea recipe PR #2 @a0f2db8 (app.ini seed-on-empty into writable volume) —
M1 read-only crash gone; bluesky-pds recipe PR #4 @4987ba9 (caddy `${STACK_NAME}_app`) — warm health
200 (was 000). gitea/bluesky end-to-end canonical advance is operator-merge-gated (fix proven by
chaos-deploy; published tags don't carry it pre-merge) — consistent with "nothing merged", not a shrug.

### Evidence addendum 2026-07-08 (re: F-redfix-3) — discourse sha pins have rotted; verify by CONTENT

The discourse shas cited below (`9ff5e19`, `53ba0910`) **no longer resolve on the mirror**. A *later* phase
force-pushed and extended the shared branch `discourse-official-image` (now `ede6399` = `refs/pull/5/head`;
redfix's PR is no longer `#4`). This is branch drift by later work, **not** a retraction of the redfix fix,
and it does not disturb the M2 PASS — which was given against the shas that existed on 2026-06-18. The other
three recipe pins and the harness pin are exact and still resolve: mattermost-lts `ci/pg-restore`@`4ca7f418`,
gitea `ci/app-ini-writable`@`a0f2db88`, bluesky-pds `ci/warm-routing-alias`@`4987ba91`, cc-ci
`redfix-m2-harness`@`07fc6d4a`.

Historical sha lines below are left **unedited on purpose** — they record what was verified when. Use this
durable **content assertion** instead of the shas to re-verify discourse at any future head:

    git clone https://git.autonomic.zone/recipe-maintainers/discourse && cd discourse
    git show origin/discourse-official-image:compose.yml | grep -m1 'image:.*discourse'
    git show origin/discourse-official-image:compose.smtpauth.yml | grep -c sidekiq

EXPECTED: `image: discourse/discourse:3.5.3` (the official-image migration = M2's claim) and `0` (the
F-redfix-1 orphaned-sidekiq removal). Both re-confirmed by the Builder at `ede6399` on 2026-07-08, and by the
Adversary at `ede6399` and `refs/pull/4/head`@`0c4539b7`. (Caveat when reproducing absence of a sha: a
`--filter=blob:none` clone and `git fetch origin <sha>` both yield false "absent" signals — test reachability
from all `refs/heads/*` + `refs/pull/*/head` instead.)

---

## Phase: M1 — investigate + isolate + classify (IN PROGRESS)

Bootstrapped 2026-06-17T23:20Z. cc-ci healthy, no run in flight, next scheduled sweep 2026-06-21
(3-day clear window). Disk `/` 38G free (75% used).

### Isolation harness (how I reproduce each failure ALONE)

Each canon-sweep per-recipe run is `runner/nightly_sweep.run_on_tag(recipe, latest)`:
`abra.recipe_checkout(recipe, <latest-tag>)` then `run_recipe_ci.py` with `RECIPE=<r>
CCCI_SKIP_FETCH=1` and REF/QUICK/MODE/VERSION unset (cold, full, head==tag). Isolation = run ONE
recipe at a time with NO concurrent sweep load on the single node (the loaded node is the known
flake source per phase plan §2.1). Runs execute on cc-ci from `/etc/cc-ci`.

### Starting canonical state (cc-ci `/var/lib/ci-warm/<r>/canonical.json`, read 2026-06-17T23:19Z)

| Recipe | Canonical now | Note |
|---|---|---|
| discourse | (none) | no canonical dir |
| mattermost-lts | (none) | no canonical dir |
| mumble | `1.0.0+v1.6.870-0` @ 20260617T180501Z | **canonical PRESENT, written TODAY** — flake signal |
| bluesky-pds | (none) | no canonical dir |
| gitea | `3.5.3+1.24.2-rootless` @ 20260617T083930Z | 3.6.0 advance not promoted |
| keycloak | (none) | de-enrolled (WARM_CANONICAL off) |

### M1 investigation tracker

| Recipe | Isolation run | Result | Root cause | Classification |
|---|---|---|---|---|
| discourse | DONE @23:40Z (`/tmp/redfix-discourse.log` on cc-ci) | install/backup/restore/custom PASS; **upgrade overlay FAIL**. Deploys+serves fine — NOT a timeout/FATA. | cc-ci overlay `tests/discourse/test_upgrade.py` asserts head runs official `discourse/discourse:3.5.3` + drops sidekiq; latest tag `0.8.1+3.5.0` AND main both still `bitnamilegacy/discourse:3.5.0`+sidekiq (migration exists in no release/main). The `depends_on discourse` string is a non-fatal prepull-only warning, not the deploy. | **stale/PR-specific cc-ci OVERLAY test** mismatched to canonical-sweep context (not flake/timeout/recipe-deploy/warm-machinery) |
| mattermost-lts | DONE @00:05Z (`/tmp/redfix-mattermost-lts.log`) | install/upgrade/backup/custom PASS; **restore FAIL** `ci_marker does not exist` — **deterministic in isolation** (not a load race) | recipe `postgres` svc backup labels: backs up hot live PGDATA + dump but has **NO `backupbot.restore.post-hook`** to replay the dump → restore doesn't round-trip postgres. Contrast immich (passes): dump-only `backup.volumes.postgres.path: backup.sql` + `restore.post-hook: /pg_backup.sh restore`. | **genuine RECIPE defect** at latest → recipe PR (adopt immich-style dump+restore-post-hook) |
| mumble | DONE — **2× isolation GREEN** (`/tmp/redfix-mumble.log` + `/tmp/redfix-mumble2.log`) | **ALL tiers PASS** incl. handshake on BOTH runs; no orphans; canonical re-promoted green each time | handshake (TLS+ServerSync) not completing within ~60s retry under heavy concurrent sweep load; fine in isolation | **load/timing FLAKE** → harness stabilization (readiness gate / retry) |
| bluesky-pds | DONE @00:45Z (`/tmp/redfix-bluesky-pds.log` + live diag) | cold lifecycle GREEN; **WC5 promote 000** reproduces (warm /xrpc/_health last status 0). NOT a flake | caddy on-demand TLS (`ask http://app:3000/tls-check`) can't reach app: caddy resolves bare `app` to OTHER stacks' app endpoints on shared `proxy` net (getent app→only 10.10.0.X, never internal 10.0.3.3; proxy has drone/traefik/keycloak/ccci `app` aliases) → no cert → 000. Promote machinery correct (refused to write canonical). | **genuine routing/RECIPE defect** (cross-stack `app`-alias collision on shared proxy) → recipe PR: unique PDS service name/alias. NOT promote-machinery, NOT flake |
| gitea | DONE @00:14Z (`/tmp/redfix-gitea2.log` + live container logs) | cold lifecycle (incl fresh 3.5.3→3.6.0 upgrade) PASS; **warm advance crash-loops** | `LoadCommonSettings() [F] error saving JWT Secret … failed to save "/etc/gitea/app.ini": read-only file system` — gitea 3.6.0/1.24.2 tries to persist a JWT to the read-only app.ini docker-config mount on warm reattach (before DB migration; 3.5.3 data intact). Cold passes (fresh secrets, no rewrite). | **genuine RECIPE defect** (3.6.0 + read-only app.ini config mount on advance) → recipe PR: render app.ini into the writable config volume. (1st gitea run hit a nixenv "already deployed" leftover confound — fixed by undeploying to idle then re-running) |
| keycloak | DONE @01:05Z (code-verified; no run) | de-enrolled. `canonical_domain("keycloak")` == `WARM_DOMAINS["keycloak"]` == `warm-keycloak.ci.commoninternet.net` EXACTLY (canonical.py:42, warm.py:27,44). Live keycloak 200 /realms/master. | data-warm canonical domain uses same `warm-<r>` scheme as the live-warm OIDC provider → promote would collide with live shared SSO. No collision-free canonical namespace exists. | **HARNESS defect** (warm-domain namespace collision) → fix: collision-free `canonical_domain` for live-warm providers (`warm-canon-<r>`), then enroll keycloak |

### M1 results table (recipe → failure → isolation result → root cause → classification → fix approach)

| Recipe | Canon-sweep failure | Isolation result | Flake or genuine | Root cause | Class | Fix approach (M2) |
|---|---|---|---|---|---|---|
| discourse | "cold-deploy timeout / deploy FATA" | install/backup/restore/custom GREEN; **upgrade overlay RED** | **genuine (deterministic)** — but the canon root-cause was WRONG (no timeout, no deploy FATA) | cc-ci overlay `tests/discourse/test_upgrade.py` asserts head = official `discourse/discourse:3.5.3` + sidekiq dropped; that migration is in NO release tag and NOT in main (all use `bitnamilegacy/discourse:3.5.0`+sidekiq) | **stale/PR-specific cc-ci OVERLAY test** | make the overlay assert migration-faithfulness only when the head IS that migration (not vs a release tag), OR a recipe PR migrating off deprecated bitnamilegacy — settle in M2 (NOT a test-weakening) |
| mattermost-lts | `test_restore_returns_state` RED | install/upgrade/backup/custom GREEN; **restore RED** (`ci_marker does not exist`) | **genuine (deterministic in isolation)** — NOT the canon "loaded-node race" | recipe postgres backup labels back up hot PGDATA + a dump but have **no `backupbot.restore.post-hook`** to replay it; restore doesn't round-trip. immich (passes) uses dump-only path + `restore.post-hook` | **genuine RECIPE defect** | recipe PR: adopt immich-style postgres dump + `backupbot.restore.post-hook` replay |
| mumble | `test_handshake…` RED | **ALL tiers GREEN** in isolation (×N) incl. handshake | **FLAKE (load/timing)** | handshake (TLS+ServerSync) doesn't complete within the 60s retry under heavy concurrent sweep load; fine isolated; canonical written green today | **load/concurrency FLAKE** | harness stabilization: stronger readiness gate before the custom tier / longer-or-smarter handshake retry |
| bluesky-pds | warm promote `/xrpc/_health` → 000 | cold lifecycle GREEN; **warm promote 000 reproduces** | **genuine (deterministic)** — NOT a load/rate-limit flake | caddy on-demand TLS calls `http://app:3000/tls-check`; caddy resolves bare `app` to OTHER stacks' app endpoints on the shared `proxy` net (every stack aliases its main svc `app`), never bluesky's own internal app (10.0.3.3) → connection refused → no cert → 000 | **genuine ROUTING/RECIPE defect** (cross-stack `app`-alias collision) | recipe PR: give the PDS service a unique name/alias so caddy resolves only bluesky's app |
| gitea | 3.5.3→3.6.0 warm advance doesn't promote | cold (incl fresh upgrade) GREEN; **warm advance crash-loops** | **genuine (deterministic)** | gitea 3.6.0/1.24.2 saves a JWT secret to `/etc/gitea/app.ini` on warm reattach; app.ini is a **read-only docker-config mount** → `read-only file system` FATA at LoadCommonSettings (pre-migration; 3.5.3 data intact). Cold passes (fresh secrets, no rewrite) | **genuine RECIPE defect** | recipe PR: render app.ini into the writable `config:/etc/gitea` volume (entrypoint) instead of a read-only docker config |
| keycloak | de-enrolled (not tested) | code-verified (no run) | **genuine (structural)** | `canonical_domain("keycloak")` == `WARM_DOMAINS["keycloak"]` == `warm-keycloak.ci.commoninternet.net` EXACTLY → a data-warm canonical would collide with the live-warm OIDC provider | **HARNESS defect** (warm-domain namespace collision) | harness: collision-free `canonical_domain` for live-warm providers (`warm-canon-<r>`), then enroll keycloak (WARM_CANONICAL=True) |

### HOW the Adversary cold-verifies each classification (run ONE recipe at a time, no concurrent load)

Isolation invocation (per recipe `R` at latest tag `T`), from `/etc/cc-ci` on cc-ci:
`git -C ~/.abra/recipes/R checkout -f --quiet T && env -u REF -u CCCI_QUICK -u MODE -u VERSION RECIPE=R CCCI_SKIP_FETCH=1 cc-ci-run runner/run_recipe_ci.py`
Latest tags: discourse `0.8.1+3.5.0`, mattermost-lts `2.1.9+10.11.15`, mumble `1.0.0+v1.6.870-0`, bluesky-pds `0.3.0+v0.4.219`, gitea `3.6.0+1.24.2-rootless`.

- **discourse** — EXPECT install/backup/restore/custom pass, upgrade fail on `test_head_runs_official_image_not_bitnamilegacy` + `test_sidekiq_service_dropped_by_head`. Confirm the overlay mismatch statically: `git -C ~/.abra/recipes/discourse show 0.8.1+3.5.0:compose.yml | grep -A1 '  app:'` and `... show main:compose.yml` both = `bitnamilegacy/discourse:3.5.0`; `grep -c 'sidekiq:'` = 1 in both. So the test's `discourse/discourse:3.5.3`/no-sidekiq expectation exists nowhere upstream.
- **mattermost-lts** — EXPECT restore fail `relation "ci_marker" does not exist`. Confirm root cause statically: `git -C ~/.abra/recipes/mattermost-lts show 2.1.9+10.11.15:compose.yml | grep backupbot` shows pre-hook + `backup.path` but NO `restore.post-hook`; immich `git -C ~/.abra/recipes/immich show <latest>:compose.yml | grep backupbot` shows `restore.post-hook: /pg_backup.sh restore`.
- **mumble** — EXPECT all tiers green (run 2–3× to confirm reproducibly green isolated). Canonical written green: `cat /var/lib/ci-warm/mumble/canonical.json`.
- **bluesky-pds** — EXPECT cold green, WC5 promote `!! WC5 promote failed … warm-bluesky-pds … last status 0`. While the warm stack is up, confirm root cause: caddy logs `dial tcp 10.10.0.X:3000: connect: connection refused` for `app:3000/tls-check`; `docker exec <caddy> getent hosts app` returns proxy IPs (10.10.0.X), the app's real internal IP is 10.0.3.x; `docker network inspect proxy | grep _app` shows many stacks aliasing `app`. (Tear down the orphaned warm-bluesky-pds stack + volumes after.)
- **gitea** — REQUIRES idle canonical first: if warm-gitea is deployed, `docker stack rm warm-gitea_ci_commoninternet_net` (retains data+config volumes) so the advance reattaches from idle. EXPECT cold green, warm advance crash-loop with container log `LoadCommonSettings() [F] error saving JWT Secret … "/etc/gitea/app.ini": read-only file system`. Restore: leave warm-gitea undeployed (idle 3.5.3, volumes retained) — registry stays `3.5.3+1.24.2-rootless`.
- **keycloak** — no run. Code-verify: `canonical.canonical_domain('keycloak')` → `warm.stable_domain('keycloak')` → `warm-keycloak.ci.commoninternet.net`; `warm.WARM_DOMAINS['keycloak']` == same string (runner/harness/canonical.py:42-44, warm.py:27-29,44-48). Live keycloak 200 on `/realms/master`.

### Node state left clean
All isolation runs torn down; orphaned warm-bluesky-pds stack+volumes removed; warm-gitea restored to idle 3.5.3 (volumes retained, registry unchanged); only live warm-keycloak deployed (healthy). No `run_recipe_ci.py` processes.

## M1 — PASS @ 2026-06-18T01:18Z (REVIEW-redfix.md; all 6 classifications cold-verified CORRECT by Adversary's own isolation re-runs). No VETO. Cleared to M2.

## Phase: M2 — FIX + verify all six (IN PROGRESS)

Fix designs locked in BACKLOG-redfix.md. Recipe PRs (mattermost-lts/bluesky/gitea) on git.autonomic.zone
mirrors via the recipe mirror+PR flow, verified `!testme` (NEVER merge). Harness fixes (keycloak/mumble)
on a cc-ci branch, verified via the harness. discourse: overlay-scope decision. Node now free for my
deploys (Adversary done with M1).

### M2 fix tracker (updated 2026-06-18T05:53Z — ALL VERIFIED)

| Recipe | Class | Fix | PR/branch + ref | Status |
|---|---|---|---|---|
| mattermost-lts | recipe defect | pg_backup.sh + `backupbot.restore.post-hook` (immich pattern) | mirror PR #1 `ci/pg-restore` @4ca7f418 | **VERIFIED** — !testme run #901 ALL tiers green incl `test_restore_returns_state` |
| discourse | stale cc-ci overlay | recipe: bitnamilegacy->official discourse image migration + drop orphaned image-less sidekiq from compose.smtpauth.yml (F-redfix-1) | mirror PR #4 `discourse-official-image` @9ff5e19 | **VERIFIED** — own cold run `/tmp/redfix-discourse-m2verify.log` **level=5 of 5** (all tiers + lint R011 PASS); F-redfix-1 regression fixed |
| keycloak | harness defect | collision-free `canonical_domain` (`warm-canon-<r>` for WARM_DOMAINS recipes) + enroll | cc-ci branch `redfix-m2-harness` @61211db | **VERIFIED** — branch-checkout run promotes at warm-canon-keycloak; live warm-keycloak 200 throughout |
| mumble | load/timing flake | harness: handshake readiness budget 60s->180s | cc-ci branch `redfix-m2-harness` @07fc6d4 | **VERIFIED** — branch-checkout run all tiers green incl handshake; budget active+non-regressing |
| gitea | recipe defect | app.ini->staging `/etc/gitea/app.ini.init` + docker-setup seed-on-EMPTY + DOCKER_SETUP_SH_VERSION v3 | mirror PR #2 `ci/app-ini-writable` @a0f2db8 | **VERIFIED** (direct chaos-deploy; promote merge-gated — see below) |
| bluesky-pds | recipe defect (routing) | caddy `{$APP_HOST}=${STACK_NAME}_app` (operator: NO rename) + CADDYFILE_VERSION v2 | mirror PR #4 `ci/warm-routing-alias` @4987ba9 | **VERIFIED** (direct chaos-deploy; promote merge-gated — see below) |

cc-ci-side change verification: run from a checkout of `redfix-m2-harness` (CCCI_REPO=<checkout>);
never touches /etc/cc-ci main. `redfix-m2-harness` is now mumble+keycloak ONLY (bluesky needs no
cc-ci change with the ${STACK_NAME}_app approach; the rename's exec-ref commit b96b8a4 was dropped).

## Gate: M2 — RE-CLAIMED, awaiting Adversary (2026-06-18T06:55Z; orig claim 05:53Z)

**Re-claim delta (addresses Adversary M2 FAIL @06:42Z — finding F-redfix-1).** The first M2 verdict was
FAIL on discourse ONLY (other 5 PASS, do-not-redo). F-redfix-1: the official-image migration dropped
`sidekiq` from compose.yml but left a dangling image-less `sidekiq:` block in `compose.smtpauth.yml` →
L5 lint R011 fail (run level=4) + broken SMTP-auth deploy. **FIXED** in PR #4 `discourse-official-image`
@**9ff5e19** (force-pushed onto @53ba0910): dropped the orphaned `sidekiq:` block; the `app:` override
already carries `DISCOURSE_SMTP_PASSWORD_FILE` + `smtp_password` secret (sidekiq is internal to the
official image), so no SMTP coverage lost. `grep sidekiq compose*.yml` = 0.
**VERIFIED two ways:** (1) the Adversary's exact lint.py repro flow at 9ff5e19 → **R011 ✅**; (2) my own
full cold run `/tmp/redfix-discourse-m2verify.log` → `RUN SUMMARY ... level=5 of 5`, all tiers pass
(install/upgrade/backup/restore/custom), `lint rung: pass`. Node clean: no discourse stack, NO discourse
canonical (untagged migrated head correctly does not promote — should_promote tagged-gate), recipe reset
to published tag 0.8.1+3.5.0. The other 5 fixes are unchanged since their Adversary PASS (keycloak,
mumble, gitea, bluesky-pds, mattermost-lts) — no re-run needed.

Adversary cold-verify for discourse: clone discourse @9ff5e19, run `RECIPE=discourse CCCI_SKIP_FETCH=1
… run_recipe_ci.py` → EXPECT level=5 of 5 (lint R011 ✅, all tiers pass, both upgrade-overlay tests
`test_head_runs_official_image_not_bitnamilegacy` + `test_sidekiq_service_dropped_by_head` pass); OR the
lint-only repro in F-redfix-1 → R011 ✅. `grep -c sidekiq ~/.abra/recipes/discourse/compose*.yml` @9ff5e19 = 0.

---
## Gate: M2 — original claim (2026-06-18T05:53Z)

**WHAT (M2 DoD).** All six canon-sweep failures FIXED — each via a recipe PR or a harness improvement —
and verified green. No recipe left as a standing exception. Nothing merged (operator merges). Per recipe:

- **mattermost-lts** (recipe PR #1) — added `pg_backup.sh` + postgres `backupbot.restore.post-hook` so
  the logical dump round-trips on restore.
- **discourse** (recipe PR #4) — migrated the head off deprecated `bitnamilegacy` to the official
  `discourse/discourse` image so the stale PR-faithfulness overlay (`test_head_runs_official_image…`,
  `test_sidekiq_service_dropped…`) passes on the migrated head (NOT a test-weakening).
- **keycloak** (harness branch) — `canonical_domain` returns a collision-free `warm-canon-<r>` for
  recipes in `warm.WARM_DOMAINS` (live-warm OIDC providers); keycloak enrolled (WARM_CANONICAL=True).
- **mumble** (harness branch) — handshake readiness budget widened 60s->180s (load-flake stabilization).
- **gitea** (recipe PR #2) — app.ini is now seeded into the WRITABLE `/etc/gitea` volume by
  docker-setup (`if [ ! -s /etc/gitea/app.ini ]`, seed-on-EMPTY) from the read-only staging config
  `app.ini.init`; `DOCKER_SETUP_SH_VERSION` v1->v3 forces the new docker-setup to re-mount. Gitea
  1.24.2 can then persist its JWT secret (the M1 read-only-app.ini crash is gone).
- **bluesky-pds** (recipe PR #4) — caddy resolves its OWN app via the fully-qualified swarm name
  `${STACK_NAME}_app` (caddy `{$APP_HOST}` env, set in the caddy service) instead of bare `app`, which
  collided with other stacks' `app` aliases on the shared `proxy` net. CADDYFILE_VERSION v1->v2.

**HOW + EXPECTED + WHERE (Adversary cold-verify, one recipe at a time, no concurrent load):**

- **mattermost-lts** — read-only artifact: `/var/lib/cc-ci-runs/901/` on cc-ci — all tiers pass,
  `junit/restore__cc-ci__test_restore.xml` testsuite failures=0, `test_restore_returns_state` pass.
  OR re-run !testme on PR #1 @4ca7f418. EXPECT restore green.
- **discourse** — !testme on PR #4 @53ba0910 (run #849 green) OR run from a checkout of the migrated
  head: EXPECT install/backup/restore/custom + upgrade overlay all pass (head now official image).
- **keycloak** — from a `redfix-m2-harness` @61211db checkout (CCCI_REPO=<checkout>), run
  `RECIPE=keycloak CCCI_SKIP_FETCH=1 ... run_recipe_ci.py`. EXPECT all cold tiers pass + WC5 promote
  succeeds at domain `warm-canon-keycloak.ci.commoninternet.net` (NOT warm-keycloak); live
  `warm-keycloak.ci.commoninternet.net/realms/master` stays 200 throughout. Code: `canonical.py`
  canonical_domain returns warm-canon-<r> for r in warm.WARM_DOMAINS.
- **mumble** — from `redfix-m2-harness` @07fc6d4 checkout, run `RECIPE=mumble CCCI_SKIP_FETCH=1 …`.
  EXPECT all 5 tiers green incl `custom/test_protocol_handshake.py::test_handshake_completes_with_
  channel_presence`; handshake budget = 36 attempts / 180s (was 60s). (Load-flake is not
  deterministically reproducible; this verifies the stabilization is applied, sound, non-weakening.)
- **gitea** (recipe PR #2 @a0f2db8 on mirror branch `ci/app-ini-writable`) — DIRECT chaos-deploy proof
  (the harness WC5 promote is merge-gated, see NOTE). With the idle 3.5.3 canonical present:
  `cd ~/.abra/recipes/gitea && git checkout -f a0f2db8` then chaos-deploy onto the retained canonical
  volumes (0-byte app.ini = genuine pre-fix 3.5.3 state):
  `abra app deploy warm-gitea.ci.commoninternet.net -C -o -n`. EXPECT: service 1/1; the config volume's
  `app.ini` seeded 0->~1862 bytes (`INSTALL_LOCK = true`); `/api/v1/version` -> 200 {"version":"1.24.2"}
  and `/api/healthz` -> 200 (curl inside the app container); retained 3.5.3 data adopted (data dirs
  dated 2026-06-17T08:39); ZERO `read-only file system` crashes in `docker service logs` (M1 crashed
  here). Evidence: `/tmp/redfix-gitea-m2-directproof.log` on cc-ci. Teardown: `abra app undeploy … -n`,
  truncate the volume app.ini to 0 (restore pre-fix state). canonical.json stays 3.5.3 idle e6a1cc79.
- **bluesky-pds** (recipe PR #4 @4987ba9 on mirror branch `ci/warm-routing-alias`) — DIRECT chaos-deploy
  proof (warm-promote is the only failing path; merge-gated). `git checkout -f 4987ba9`; generate
  secrets (`abra app secret generate warm-bluesky-pds.ci.commoninternet.net --all -m -C -o -n`) + insert
  a PLC rotation key (tests/bluesky-pds/install_steps.sh logic: 32-byte hex into pds_plc_rotation_key
  v1); **re-checkout 4987ba9 AFTER secret ops** (abra secret insert force-fetches+reverts the checkout);
  `abra app deploy warm-bluesky-pds.ci.commoninternet.net -C -o -n` (EXPECT `caddyfile: v1 -> v2`,
  NEW DEPLOYMENT 4987ba9). EXPECT: app+caddy 1/1; inside caddy `getent hosts
  warm-bluesky-pds_ci_commoninternet_net_app` -> a 10.0.x.x INTERNAL ip (own stack) while
  `getent hosts app` -> a 10.10.x.x proxy ip (foreign, the M1 collision); caddy log "certificate
  obtained successfully" with 0 "connection refused"; external `curl https://warm-bluesky-pds.ci.
  commoninternet.net/xrpc/_health` -> **200** {"version":"0.4.219"} (M1 was 000). Evidence:
  `/tmp/redfix-bluesky-m2-directproof.log`. Teardown: undeploy + remove volumes (caddy_data, pds_data)
  + secrets (no canonical, matching M1).

**NOTE — gitea & bluesky end-to-end canonical-promote is OPERATOR-MERGE-GATED (not a shrug).** The
harness WC5 promote does a recipe_checkout(published-tag)+non-chaos deploy, and BOTH run_recipe_ci.py:373
AND abra force-fetch `refs/tags/*` from upstream (abra.py:135 documents this), so any local move of the
release tag to the fix commit is reverted to the PUBLISHED commit. The published 3.6.0 / 0.3.0 tags do
NOT yet carry the fix (PR not merged — operator merges, per phase guardrail), so pre-merge the promote
necessarily deploys the unfixed published release. Confirmed empirically: a full gitea harness run's WC5
promote deployed 357926f and crash-looped exactly like M1. The DIRECT chaos-deploy (chaos = deploy the
working-tree checkout = the PR fix) is therefore the MAXIMAL + faithful pre-merge proof — it reproduces
the EXACT M1 failing scenario (gitea: the retained canonical volumes; bluesky: warm-bluesky-pds on the
shared proxy) and shows the fix resolves it. End-to-end canonical advance follows automatically once the
operator merges PR #2 / #4 and the release tag carries the fix. This is NOT a standing exception — the
defect is fixed + proven; only the registry-advance awaits the operator's merge (the phase's own
"nothing merged" constraint).

**WHERE (refs).** Recipe PRs on `git.autonomic.zone/recipe-maintainers/<recipe>`: mattermost-lts
`ci/pg-restore`@4ca7f418, discourse `discourse-official-image`@53ba0910, gitea `ci/app-ini-writable`
@a0f2db8, bluesky-pds `ci/warm-routing-alias`@4987ba9. cc-ci harness branch
`redfix-m2-harness`@07fc6d4 (keycloak 61211db + mumble 07fc6d4). Reasoning/dead-ends in
JOURNAL-redfix.md. Node left clean (only infra + live warm-keycloak 200; gitea idle 3.5.3 volumes
retained, canonical e6a1cc79 unchanged; no bluesky/test stacks/volumes/secrets; no run procs).

## Gate: M1 — PASS (above).

**WHAT (M1 DoD).** All six canon-sweep failures investigated in ISOLATION (one recipe at a time, no
concurrent sweep load), root-caused with first-hand evidence, and classified (flake vs genuine; recipe
vs test vs warm-machinery vs load) — see the **M1 results table** + **HOW the Adversary cold-verifies**
sections above. Summary: discourse = stale cc-ci overlay test (canon timeout/FATA root-cause was
wrong); mattermost-lts = genuine recipe defect (no `backupbot.restore.post-hook`); mumble = load/timing
FLAKE (2× isolation green); bluesky-pds = genuine routing defect (caddy↔app `app`-alias collision on
shared proxy); gitea = genuine recipe defect (read-only app.ini config mount + 3.6.0 JWT save);
keycloak = harness warm-domain namespace collision. NO "probably a flake" — every classification has
an isolation re-run or code proof.

**HOW + EXPECTED + WHERE.** Per-recipe cold-verify commands, expected outputs, and evidence paths are
in the two sections above ("M1 results table" and "HOW the Adversary cold-verifies each classification").
Evidence logs on cc-ci: `/tmp/redfix-{discourse,mattermost-lts,mumble,mumble2,bluesky-pds,gitea2}.log`.
Reasoning/dead-ends in JOURNAL-redfix.md. Node left clean (see "Node state left clean" above).

## Untracked file `main.go` — NOT produced by this phase (operator: unexplained)

**WHAT.** An untracked `main.go` (281 bytes, a Go "Hello, World!" `net/http` server on `:8080`) sits in the
repo root. It is not in any commit on any ref, is not gitignored, and is unrelated to every redfix DoD item.
No `go.mod` accompanies it. I did not create it and have not committed, modified, or deleted it.

**WHERE.** `/srv/cc-ci-orch/cc-ci/main.go` — **one file in one clone.**

**RETRACTED (2026-07-09, Adversary A-redfix-3):** an earlier version of this entry claimed the file was
"present in **both** clones … i.e. written by something outside git." **That evidence is void.** `/srv/cc-ci`
is a **symlink to `/srv/cc-ci-orch`**, so `/srv/cc-ci/cc-ci/main.go` and `/srv/cc-ci-orch/cc-ci/main.go` are
the **same inode** — one file seen twice, not two clones. The "a git operation cannot leave the same untracked
file in two clones" inference therefore carries **no information** about the file's origin, which remains
**unexplained**. Independently re-derived: `ls -la /srv/` shows `cc-ci -> /srv/cc-ci-orch`, and
`stat -c '%i %h %n'` on both paths prints the same inode `3254604` with `links=1`.

**HOW to verify.** `git log --all --oneline -- main.go` → empty (in no commit).
`git status --short` → `?? main.go`.
`ls -la /srv/ | grep cc-ci` → `cc-ci -> /srv/cc-ci-orch` (the symlink that voids the "both clones" claim).
`stat -c '%i %h %n' /srv/cc-ci/cc-ci/main.go /srv/cc-ci-orch/cc-ci/main.go` → same inode, `links=1`.

**Risk: inert** (Adversary-confirmed). 281-byte hello-world `net/http` listener; no `go.mod`; `go` is **not
installed**; **nothing listening on `:8080`**; in no commit on any ref; unreferenced by any tracked file.

**EXPECTED.** Left in place, untracked. Per guardrails I do not delete or commit files I did not create;
flagging for the operator rather than acting. It affects no gate, no DoD item, and no verdict.

## Blocked

(none)