# Phase `redfix` — investigate every canon-sweep failure + open fix PRs

**Mission (operator-specified 2026-06-17):** the `canon` cold-full-lifecycle sweep (every recipe's latest
release deployed from scratch + warm-promote) surfaced failures the **upgrade-only weekly run never
tests**. Investigate **all** of them, root-cause each, and **FIX every one** — each via **a recipe PR or a harness
improvement** (operator 2026-06-17). **None is left as a standing exception** (keycloak's de-enrollment and
gitea's "documented limitation" are NOT acceptable end states — they must be fixed). The operator recalls
**mattermost-lts and mumble passing before**, so **flake-vs-genuine classification (via isolated re-runs)
is the crux** — but note a flake still gets a fix (a harness stabilization improvement), not a shrug.

State files: `STATUS-redfix.md`, `BACKLOG-redfix.md`, `REVIEW-redfix.md`, `JOURNAL-redfix.md`. DECISIONS.md shared.

## 1. The failing set (from canon DECISIONS §canon exceptions)

| Recipe | What happened in the sweep | First hypothesis |
|---|---|---|
| **discourse** | cold-deploy **timeout** (rc=142/143, ~51-min wedge) | slow Rails boot under load → timeout/headroom, maybe the start-first rollout |
| **mattermost-lts** | `test_restore.py::test_restore_returns_state` FAILED at latest | restore-tier — possibly the known backup/restore db-cycle race on the loaded node |
| **mumble** | `custom/test_protocol_handshake.py::test_handshake_completes_with_channel_presence` FAILED | readiness/timing of the protocol handshake |
| **bluesky-pds** | cold test **GREEN**, warm-canonical promote FAILED (`warm-bluesky-pds…` → 000, cold domain + other 15 warm domains route fine) | warm-canonical-deploy **routing machinery** (cc-ci side) |
| **gitea** | cold test **GREEN**, `3.5.3→3.6.0` warm **advance** doesn't promote (app.ini read-only) | recipe: read-only `app.ini` blocks in-place warm advance (ties to LFS PR #1) |
| **keycloak** | **de-enrolled** (not tested) — live-warm OIDC dep collision on `warm-keycloak…` | **harness fix:** canonical warm deploys need a domain/namespace that can't collide with a live service, so keycloak can enroll |

## 2. Method — per recipe: isolate → root-cause → classify → fix-PR

1. **Reproduce in ISOLATION first.** Re-run each failing recipe through the harness **alone** (not under
   concurrent sweep load) on cc-ci. The loaded single node during the sweep is a known flake source
   (backup/restore db-cycle races, deploy timeouts). This directly answers the operator's "they passed
   before": if a recipe is **green in isolation**, it's a load/concurrency **flake**, not a recipe defect
   — and the fix is harness stabilization (or a documented known-flake), NOT a recipe change.
2. **Root-cause** with real evidence — the actual assertion/error, deploy logs (`docker service logs`),
   the wedge cause. No guessing.
3. **Classify** each precisely:
   - **Recipe defect at latest** → a **recipe PR** on the `git.autonomic.zone` mirror via the
     `recipe-upgrade`/`recipe-upstream` flow, verified with `!testme` (NEVER merge — operator).
   - **Stale/wrong cc-ci test** → per the standing discipline: leave a PR **comment**, or a cc-ci
     **test PR** only under the `--with-tests` gate. **Never weaken a test** to turn a red green.
   - **Warm-canonical machinery defect** (bluesky routing) → a **cc-ci branch PR** (single-writer; never
     push main).
   - **Load/concurrency flake** → a harness **stabilization** fix (timeout/serialization/readiness/retry)
     as a cc-ci PR, or a documented known-flake if genuinely environmental.
4. **Open the fix PR** for each and **verify it** (recipe PR green via `!testme`; cc-ci PR via the
   harness). One PR per fix; capture URLs.

## 3. Per-recipe specifics to chase

- **discourse** — characterise the timeout/wedge: is it just headroom (raise `DEPLOY_TIMEOUT` / give it an
  uncontended deploy) or a real convergence bug? It cold-deploys fine in the weekly with headroom. Fix =
  cc-ci timeout/serialization tuning and/or a recipe convergence fix — whichever the evidence shows.
- **mattermost-lts** — does `test_restore_returns_state` fail in isolation? If green → the loaded-node
  restore race (cf. plausible/ghost) → stabilize (e.g. a `BACKUP_VERIFY`-style settle/retry). If red →
  diagnose the restore (recipe vs test) and fix the right side.
- **mumble** — does the handshake test fail in isolation? Likely a readiness/timing gap (handshake before
  the channel is up) → readiness probe / retry, or a genuine recipe/test issue.
- **bluesky-pds** — why does `warm-bluesky-pds…` return 000 while the cold-test domain and the other 15
  warm domains route fine? Find the warm-deploy routing defect (cc-ci warm machinery) and fix it so
  bluesky promotes; it should then drop from the exception list.
- **gitea** — **fix it** so it promotes `3.6.0`: either a **recipe PR** making `app.ini` writable for the
  in-place warm advance, OR a **harness improvement** so the warm advance falls back to a clean re-deploy
  when an in-place config rewrite isn't possible. Pick per evidence; the bar is gitea promotes `3.6.0`
  (ties to the LFS `app.ini` change in PR #1). Not "documented as inherent."
- **keycloak** — **fix via a harness improvement** so it can be enrolled safely: canonical warm deploys
  must use a domain/namespace that can **never** collide with a live shared service (e.g. a dedicated
  `warm-canon-<recipe>` / per-canonical namespace), so keycloak's canonical deploys without touching the
  live OIDC service. (Likely shares a root cause with bluesky's warm-routing — a single warm-domain fix may
  resolve both.) Don't risk the live keycloak — the fix is isolation, then enroll.

## 4. Gates

**M1 — all investigated, isolated, classified.** Every recipe in §1 re-run in isolation with evidence; a
results table `recipe → failure → isolation result (flake|genuine) → root cause → classification → fix
approach`. Adversary cold-verifies the classifications: a claimed **flake** is reproducibly green in
isolation (and red under load); a claimed **recipe defect** is genuinely the recipe (not a stale test or a
harness artifact); a claimed **warm-machinery** bug is in cc-ci, not the recipe. No "it's probably a
flake" without an isolation re-run proving it.

**M2 — all six FIXED + verified.** For each, the appropriate fix (a recipe PR or a harness improvement)
exists and is **verified green** (recipe PR via `!testme`; harness/cc-ci PR via the harness;
flake-stabilization re-run green under load). **All six promote/pass after the fix** — bluesky promotes
(warm-routing fixed), gitea promotes `3.6.0`, keycloak enrolled + promotes via the collision-free warm
domain, discourse converges in time, mattermost-lts + mumble green (stabilized if flake, fixed if real).
**No recipe left as a standing exception.** If one genuinely cannot be fixed at the recipe level, the fix
moves to the harness (or a recipe PR + a tracked upstream issue) — never a shrug. **Nothing merged**
(operator merges). A short report: per recipe, flake-or-real, the fix, and the verification. Fresh
Adversary PASS on both milestones → `## DONE`.

## 5. Guardrails

- **Isolation before blame.** Never fix a load-flake as a recipe bug, or a recipe bug as a flake — the
  isolation re-run is mandatory evidence (especially for mattermost-lts/mumble, which the operator saw pass).
- **Recipe mirrors are PR-only** — recipe defects get a PR, `!testme`-verified, **never merged**. **Never
  weaken a cc-ci test** to make a recipe green; a stale test gets a comment or a `--with-tests` test PR.
- **cc-ci changes** on a dedicated branch (single-writer; never push main, never disturb other clones).
- **Shared swarm:** ≤2–3 concurrent deploys; tear down every deploy on every exit path; mind the live
  warm services (keycloak especially). Host changes coordinated (loops may rebuild if clean + verify health).
- Honest reporting — a flake is labelled a flake with proof; a recipe left unfixed is called out with the
  reason. Commit author `autonomic-bot <autonomic-bot@noreply.git.autonomic.zone>`; push every commit; abra
  over a pseudo-TTY.

## 6. Definition of Done

All six canon-sweep failures (discourse, mattermost-lts, mumble, bluesky-pds, gitea, keycloak)
investigated in isolation, root-caused, classified (flake vs genuine; recipe vs test vs warm-machinery vs
load), and **FIXED — each via a recipe PR or a harness improvement — and verified green**: bluesky promotes
(warm-routing fixed), gitea promotes `3.6.0`, keycloak enrolled via a collision-free warm domain, discourse
converges, mattermost-lts + mumble green (stabilized or fixed). **None left as a standing exception.**
Nothing merged (operator merges); a clear flake-vs-real report with the fix + verification per recipe.
M1 + M2 fresh Adversary PASSes in REVIEW-redfix.md.