cc-ci/machine-docs/STATUS-redfix.md

# STATUS — phase `redfix`

Phase SSOT: `/srv/cc-ci/cc-ci-plan/plan-phase-redfix-canon-sweep-failures.md`

Mission: investigate every canon-sweep failure (discourse, mattermost-lts, mumble, bluesky-pds,
gitea, keycloak) → isolate → root-cause → classify (flake vs genuine; recipe vs test vs
warm-machinery vs load) → FIX each (recipe PR or harness improvement) → verify green. No standing
exceptions. Nothing merged.

## DONE — 2026-07-09T00:18Z

Phase `redfix` COMPLETE. All six canon-sweep failures investigated in isolation, root-caused, classified,
**FIXED — each via a recipe PR or a harness improvement — and verified green**; no recipe left as a standing
exception; nothing merged (operator merges). Every gate has a fresh Adversary PASS in REVIEW-redfix.md and
there is **no standing VETO**:
- **M1 PASS** @ 2026-06-18T01:18Z (investigation/classification cold-verified).
- **M2 PASS** @ 2026-07-09T00:18Z (F-redfix-4 remedy cold-verified; VETO CLEARED; supersedes the
  2026-06-18T07:06Z M2 PASS, which was given against parent `07fc6d4`).

Adversary findings F-redfix-1/2/3/4 are all **CLOSED**; no open blocking finding.

## ⚠ OPERATOR ACTION REQUIRED — B-redfix-8 (HIGH, open, outside DoD)

Phase DoD is met, but **one HIGH item is open and only an operator can close it.** It does not block `##
DONE` (no DoD item depends on it) and is recorded in full in BACKLOG-redfix.md.

**WHAT.** The live Gitea bot password (`GITEA_PASSWORD` in `/srv/cc-ci/.testenv`) was committed in
`14c7dee` (`machine-docs/BACKLOG-redfix.md`), pushed to `origin/main`, and is served **in cleartext to the
unauthenticated public internet** by the public mirror. It is push-capable to `recipe-maintainers/*`. HEAD
and the redaction commit `e99e2b3` are clean; only the historical commit `14c7dee` serves it. Redaction
cannot unpublish it — the value is permanent in history and in every clone.

**WHERE.** `git.autonomic.zone/recipe-maintainers/cc-ci` →
`/raw/commit/14c7dee/machine-docs/BACKLOG-redfix.md`

**HOW to confirm exposure** (unauthenticated fetch; a raw blob at a fixed sha is immutable, so served size
must equal object size):

    git cat-file -s 14c7dee:machine-docs/BACKLOG-redfix.md    # → 33408

An unauthenticated fetch of the raw URL returning **HTTP 200 / 33408 bytes** is the leak, not an error page.

**HOW to confirm it is still the live credential** (commits to the value without republishing it).
Run this **on the orchestrator** — `/srv/cc-ci/.testenv` lives there, **not** on `cc-ci`:

    python3 -c 'import hashlib,re;v=re.search(r"^GITEA_PASSWORD=(.*)$",open("/srv/cc-ci/.testenv").read(),re.M).group(1).strip().strip(chr(34)).strip(chr(39));print(hashlib.sha256(v.encode()).hexdigest()[:16])'

**EXPECTED.** Prints `3fcea78925015fc9` while still unrotated. **A different digest ⇒ rotated ⇒ `14c7dee`
is inert ⇒ close B-redfix-8** — *but only if the probe was sound.* Two known-bad probes yield a different
digest without any rotation, and must NOT be used to close this item:

- **`e3b0c44298fc1c14` is the empty-input tell, not a rotation.** Running the command over `ssh cc-ci`
  fails to open `.testenv`, hashes the empty string, and prints `sha256("")[:16]` = `e3b0c44298fc1c14`
  (check: `printf '' | sha256sum`). Fix the probe; do not close B-redfix-8.
- **Do not conflate the three digests.** `3fcea78925015fc9` = sha256(credential *value*)[:16] — the only
  rotation sentinel. `1994fd8d…` = sha256(the whole served *blob*) — immutable, changes never.
  `e3b0c44298fc1c14` = sha256(empty) — a broken probe.

Exposure confirmed at **value level** (Adversary, 2026-07-09T07:34Z): the currently-valid, push-capable
password appears **verbatim** in the publicly served body, not merely "a blob that once held a secret."

**REMEDY (operator).** Rotate the Gitea bot password, then update `/srv/cc-ci/.testenv`. The credential is a
Class-A1 external infra input (plan §4.4) — the Builder must not rotate or invent it. History rewrite is not
available to the Builder either (`--force` is forbidden by the standing rules), so rotation is the only
action that actually revokes the exposure.

**Merge target: `redfix-m2-harness` @ `b5f2b10`** (not `07fc6d4` — that tip carries the F-redfix-4 defect).
The earlier `## DONE` of 2026-06-18T07:09Z was **withdrawn** on 2026-07-09 under the standing VETO
F-redfix-4 and is re-asserted here only after the remedy was Adversary-verified. Details of that cycle,
and the still-open non-blocking B-redfix-5, are below.

One deferred, non-blocking item remains recorded (NOT part of any DoD item, NOT a standing finding):
**B-redfix-5** in BACKLOG-redfix.md — in `warm_reconcile.py`, TWO post-`abra.undeploy()` calls sit outside
the upgrade's `try/except`: `warmsnap.snapshot()` on the upgrade path (`:514`) and `warmsnap.restore()` on
the rollback path (`:536`). If either raises, live keycloak is left **undeployed**. Pre-dates the
enrollment (present at `07fc6d4`). Adversary concurred it is not a VETO.

**MERGE PRECONDITION for `b5f2b10` (verify at merge time).** The F-redfix-4 fix *adds* a raise path to both
of those calls — `warmsnap._assert_slot_not_foreign()` (`warmsnap.py:158` in `snapshot()`, `:209` in
`restore()`). It raises iff the live slot's `snapshot/meta.json` records a `domain` other than the one
passed. Composed with B-redfix-5, a foreign live-slot meta wedges live keycloak: on the **upgrade** path
this fires on *every* stateful auto-upgrade, not only on rollback. It is **unreachable on cc-ci today**
because the live slot holds no snapshot at all (`read_meta` → `None` → "a slot with no snapshot yet is free
to claim"). It is unreachable for that reason — **not** because F-redfix-4 is closed.

The only state that creates a foreign live-slot meta is a **pre-fix (`07fc6d4`) canonical seed**, which
wrote the canonical's domain into the shared slot `/var/lib/ci-warm/keycloak/`. The seed is a real, not
hypothetical, arming action: `tests/keycloak/recipe_meta.py` carries `WARM_CANONICAL = True` at `07fc6d4`
too. So: do not run a canonical keycloak seed from `07fc6d4` between now and the merge.

**The wedge does not self-heal.** `reconcile()`'s only `try/except` is `warm_reconcile.py:520-525` (it wraps
`deploy_version` + `wait_healthy`), and its sole caller `main():556` does not catch either. A raise at `:514`
or `:536` therefore escapes to `SystemExit` with live keycloak already undeployed by the preceding
`abra.undeploy()` — no code path redeploys it. Recovery is manual.

**Blast radius: keycloak only.** `warm.WARM_DOMAINS == {"keycloak"}` and `keycloak` is the only `SPECS` entry
with `stateful: True` (`traefik` is `stateful: False` — version-rollback-only, never snapshots). No other
warm unit reaches either call site.

**HOW to verify the precondition** (on cc-ci; `python3` is absent there, so `ls`/`cat`):

    ssh cc-ci 'ls -A /var/lib/ci-warm/keycloak/; cat /var/lib/ci-warm/keycloak/snapshot/meta.json 2>&1'

**EXPECTED (observed 2026-07-09T08:2xZ):** `last_good` only, and `cat` reports
`No such file or directory` — no `snapshot/`, hence no `meta.json`. **A `meta.json` whose `"domain"` is
anything other than `warm-keycloak.ci.commoninternet.net` means the precondition is violated:** fix B-redfix-5
(or delete the foreign slot meta) BEFORE merging `b5f2b10`. A `meta.json` recording
`warm-keycloak.ci.commoninternet.net` is self-consistent and passes the guard.

---

### F-redfix-4 remedy (the reason DONE was withdrawn and re-asserted) — 2026-07-09

**WHAT.** F-redfix-4 is fixed at `redfix-m2-harness` @ **`b5f2b10`** (parent `07fc6d4`, the sha the M2 PASS
was given against). Warm state is now keyed by a **stack namespace**, not a bare recipe:
`canonical.canonical_ns(r)` is the single source from which BOTH the canonical's domain and its warm-state
slot derive. A live-warm provider (`r in warm.WARM_DOMAINS`) gets ns `canon-<r>` → domain
`warm-canon-keycloak.ci.commoninternet.net` (**unchanged**) + slot `/var/lib/ci-warm/canon-keycloak/`.
Every other recipe keeps ns `<r>` (the invariant is structural — `r not in warm.WARM_DOMAINS` — not a
property of any particular count). `WARM_DOMAINS == {"keycloak"}` is a singleton, so the re-key is provably
`keycloak -> canon-keycloak` and nothing else: of the 21 enrolled recipes the other 20 satisfy
`canonical_ns(r) == r`, so **zero existing canonical slots change**. On live disk that is 17 seeded
canonicals (those carrying a `canonical.json`), all of which remain in `prune_stale`'s `keep` set.

Addresses all four consequences: (1)+(2) slots disjoint, neither deployment can replace the other's
known-good; (3) the outage race is removed with the shared slot; (4) `prune_stale`'s reconciler-dir
invariant is now structural — `<recipe>/` never gains a `canonical.json`.

Also: `warmsnap._assert_slot_not_foreign()` refuses to snapshot/restore a slot recorded against a different
domain (defence in depth; fails before the destructive swap, not at the next restore). The two comments
asserting the deployments "can never touch each other" are corrected (`canonical.py`,
`tests/keycloak/recipe_meta.py`). `WARM_CANONICAL = True` is retained — keycloak stays enrolled, no
skip-guard, no silent de-enrollment.

**MIGRATION: none required.** On cc-ci `/var/lib/ci-warm/keycloak/` contains only `last_good` (the
canonical was never seeded), so no existing file changes slot. Verify: `ls -A /var/lib/ci-warm/keycloak/`
→ `last_good` only.

**WHERE.** cc-ci repo, branch `redfix-m2-harness` @ `b5f2b10`. Files: `runner/harness/warmsnap.py`,
`runner/harness/canonical.py`, `runner/warm_reconcile.py`, `runner/run_recipe_ci.py`,
`tests/keycloak/recipe_meta.py`, `tests/unit/test_warmsnap.py`, `tests/unit/test_canonical.py`.

**HOW to verify (1) — unit suite, cold clone, no docker needed.**

    git clone -q --branch redfix-m2-harness <cc-ci remote> /tmp/v && cd /tmp/v
    /nix/store/x188l04r3gfkh18gy1dpf05fv3kkrgs7-python3-3.12.8-env/bin/python3 -m pytest tests/unit -q

EXPECTED: `325 passed` (baseline at parent `07fc6d4` is `315 passed`; +10 F-redfix-4 regression tests).
The 10 include `test_live_and_canonical_slots_are_disjoint`, `test_slot_maps_1to1_to_its_stack`,
`test_registry_path_of_live_warm_provider_is_not_the_reconciler_dir`,
`test_prune_stale_deenrolled_provider_spares_reconciler_last_good`,
`test_snapshot_refuses_to_clobber_another_domains_slot`, `test_restore_refuses_a_foreign_slot`.

**HOW to verify (2) — the VETO's clearing condition, on the real node.** Non-destructive: writes only to a
scratch `CCCI_WARM_ROOT` + one throwaway docker volume. Uses the real idle `warm-canon-keycloak` stack for
the canonical side and a throwaway `warm-fakelive_…_data` volume as the live-warm stand-in (the live
`warm-keycloak` stack cannot be undeployed to snapshot it).

    ssh cc-ci
    docker volume create warm-fakelive_ci_commoninternet_net_data
    MP=$(docker volume inspect -f '{{.Mountpoint}}' warm-fakelive_ci_commoninternet_net_data); echo x > $MP/live.txt
    git clone -q --branch redfix-m2-harness <cc-ci remote> /tmp/v && mkdir -p /tmp/vw/keycloak
    echo '10.7.1+26.6.2' > /tmp/vw/keycloak/last_good
    cd /tmp/v/runner && CCCI_WARM_ROOT=/tmp/vw /nix/store/jag2131a95gw6ng7grig9pj3dn2q8vrv-python3-3.12.8-env/bin/python3 - <<'PY'
    import sys; sys.path.insert(0, ".")
    from harness import warmsnap as ws, canonical as c
    LIVE, CANON = ws.live_slot("keycloak"), c.canonical_slot("keycloak")
    CANON_DOM, LIVE_DOM = c.canonical_domain("keycloak"), "warm-fakelive.ci.commoninternet.net"
    print(ws.snap_dir(LIVE), ws.snap_dir(CANON), c.registry_path("keycloak"), sep="\n")
    ws.snapshot(CANON, CANON_DOM, version="canon-known-good")
    ws.snapshot(LIVE,  LIVE_DOM,  version="live-last-good")     # used to clobber the canonical
    print("restore(canon) ->", ws.restore(CANON, CANON_DOM)["volumes"])
    print("restore(live)  ->", ws.restore(LIVE,  LIVE_DOM )["volumes"])
    print("last_good:", open("/tmp/vw/keycloak/last_good").read().strip())
    try: ws.snapshot(CANON, LIVE_DOM); print("FAIL: foreign snapshot allowed")
    except ws.SnapshotError as e: print("foreign REFUSED:", str(e)[:60])
    PY
    docker volume rm warm-fakelive_ci_commoninternet_net_data; rm -rf /tmp/vw /tmp/v

EXPECTED (observed 2026-07-09):

    /tmp/vw/keycloak/snapshot                     <- live slot
    /tmp/vw/canon-keycloak/snapshot               <- canonical slot: DISJOINT
    /tmp/vw/canon-keycloak/canonical.json         <- registry NOT in the reconciler's dir
    restore(canon) -> ['warm-canon-keycloak_ci_commoninternet_net_mariadb',
                       'warm-canon-keycloak_ci_commoninternet_net_providers']
    restore(live)  -> ['warm-fakelive_ci_commoninternet_net_data']
    last_good: 10.7.1+26.6.2                      <- survived the canonical seed
    foreign REFUSED: warm slot 'canon-keycloak' holds the known-good of ...

Each `restore()` returns its OWN stack's volumes (the clearing condition). Pre-fix, the same script prints
one shared slot `/tmp/vw/keycloak/snapshot` for both and `restore(canon)` raises `SnapshotError`.

**Node integrity after my own run of the above (2026-07-09).** Real warm root untouched
(`/var/lib/ci-warm/keycloak/` = `last_good` only); `warm-canon-keycloak` mariadb digest byte-identical
across the restore round-trip (`1201440268 48846` before and after, 391 files / 2 files); live
`warm-keycloak…/realms/master` → **200** throughout; throwaway volume removed; scratch removed.

**Scope.** keycloak only. M1 classifications and the discourse / mattermost-lts / gitea / bluesky-pds /
mumble fixes are untouched by this commit and remain Adversary-verified (M1 PASS @2026-06-18T01:18Z; the
five non-keycloak M2 fixes content-verified through re-confirmation #7).

**Known residual, NOT in F-redfix-4's clearing condition** (filed as B-redfix-5 in BACKLOG-redfix.md, not
blocking): `warm_reconcile.py`'s rollback `warmsnap.restore()` still sits outside the `try/except` that
guards the upgrade, so any restore failure (e.g. a corrupt/absent snapshot) leaves live keycloak undeployed
after `abra.undeploy()`. The F-redfix-4 race that made this reachable is gone; the pre-existing robustness
gap is not. Recorded, not silently dropped.

The prior DONE text is retained verbatim below as the historical record of what was claimed on 2026-06-18.

---

## (HISTORICAL — withdrawn 2026-07-09 under VETO F-redfix-4; superseded by the DONE above) DONE — 2026-06-18T07:09Z

Phase `redfix` COMPLETE. All six canon-sweep failures investigated in isolation, root-caused,
classified, **FIXED — each via a recipe PR or a harness improvement — and verified green**; no recipe
left as a standing exception; nothing merged (operator merges). Both gates have a fresh Adversary PASS
in REVIEW-redfix.md with no standing VETO:
- **M1 PASS** @ 2026-06-18T01:18Z (investigation/classification cold-verified).
- **M2 PASS** @ 2026-06-18T07:06Z (all 6 fixes cold-verified; supersedes the 06:42Z FAIL after the
  discourse F-redfix-1 rework).

Fixes (per recipe): mattermost-lts recipe PR #1 (pg_backup.sh + restore.post-hook) — restore
round-trips; discourse recipe PR #4 @9ff5e19 (official-image migration + drop orphaned sidekiq from
compose.smtpauth.yml) — level=5, lint R011 ✅; keycloak harness (collision-free `warm-canon-<r>` +
enroll) — promotes without touching live SSO; mumble harness (handshake budget 60→180s) — flake
stabilized, non-weakening; gitea recipe PR #2 @a0f2db8 (app.ini seed-on-empty into writable volume) —
M1 read-only crash gone; bluesky-pds recipe PR #4 @4987ba9 (caddy `${STACK_NAME}_app`) — warm health
200 (was 000). gitea/bluesky end-to-end canonical advance is operator-merge-gated (fix proven by
chaos-deploy; published tags don't carry it pre-merge) — consistent with "nothing merged", not a shrug.

### Evidence addendum 2026-07-08 (re: F-redfix-3) — discourse sha pins have rotted; verify by CONTENT

The discourse shas cited below (`9ff5e19`, `53ba0910`) **no longer resolve on the mirror**. A *later* phase
force-pushed and extended the shared branch `discourse-official-image` (now `ede6399` = `refs/pull/5/head`;
redfix's PR is no longer `#4`). This is branch drift by later work, **not** a retraction of the redfix fix,
and it does not disturb the M2 PASS — which was given against the shas that existed on 2026-06-18. The other
three recipe pins and the harness pin are exact and still resolve: mattermost-lts `ci/pg-restore`@`4ca7f418`,
gitea `ci/app-ini-writable`@`a0f2db88`, bluesky-pds `ci/warm-routing-alias`@`4987ba91`, cc-ci
`redfix-m2-harness`@`07fc6d4a`.

Historical sha lines below are left **unedited on purpose** — they record what was verified when. Use this
durable **content assertion** instead of the shas to re-verify discourse at any future head:

    git clone https://git.autonomic.zone/recipe-maintainers/discourse && cd discourse
    git show origin/discourse-official-image:compose.yml | grep -m1 'image:.*discourse'
    git show origin/discourse-official-image:compose.smtpauth.yml | grep -c sidekiq

EXPECTED: `image: discourse/discourse:3.5.3` (the official-image migration = M2's claim) and `0` (the
F-redfix-1 orphaned-sidekiq removal). Both re-confirmed by the Builder at `ede6399` on 2026-07-08, and by the
Adversary at `ede6399` and `refs/pull/4/head`@`0c4539b7`. (Caveat when reproducing absence of a sha: a
`--filter=blob:none` clone and `git fetch origin <sha>` both yield false "absent" signals — test reachability
from all `refs/heads/*` + `refs/pull/*/head` instead.)

---

## Phase: M1 — investigate + isolate + classify (IN PROGRESS)

Bootstrapped 2026-06-17T23:20Z. cc-ci healthy, no run in flight, next scheduled sweep 2026-06-21
(3-day clear window). Disk `/` 38G free (75% used).

### Isolation harness (how I reproduce each failure ALONE)

Each canon-sweep per-recipe run is `runner/nightly_sweep.run_on_tag(recipe, latest)`:
`abra.recipe_checkout(recipe, <latest-tag>)` then `run_recipe_ci.py` with `RECIPE=<r>
CCCI_SKIP_FETCH=1` and REF/QUICK/MODE/VERSION unset (cold, full, head==tag). Isolation = run ONE
recipe at a time with NO concurrent sweep load on the single node (the loaded node is the known
flake source per phase plan §2.1). Runs execute on cc-ci from `/etc/cc-ci`.

### Starting canonical state (cc-ci `/var/lib/ci-warm/<r>/canonical.json`, read 2026-06-17T23:19Z)

| Recipe | Canonical now | Note |
|---|---|---|
| discourse | (none) | no canonical dir |
| mattermost-lts | (none) | no canonical dir |
| mumble | `1.0.0+v1.6.870-0` @ 20260617T180501Z | **canonical PRESENT, written TODAY** — flake signal |
| bluesky-pds | (none) | no canonical dir |
| gitea | `3.5.3+1.24.2-rootless` @ 20260617T083930Z | 3.6.0 advance not promoted |
| keycloak | (none) | de-enrolled (WARM_CANONICAL off) |

### M1 investigation tracker

| Recipe | Isolation run | Result | Root cause | Classification |
|---|---|---|---|---|
| discourse | DONE @23:40Z (`/tmp/redfix-discourse.log` on cc-ci) | install/backup/restore/custom PASS; **upgrade overlay FAIL**. Deploys+serves fine — NOT a timeout/FATA. | cc-ci overlay `tests/discourse/test_upgrade.py` asserts head runs official `discourse/discourse:3.5.3` + drops sidekiq; latest tag `0.8.1+3.5.0` AND main both still `bitnamilegacy/discourse:3.5.0`+sidekiq (migration exists in no release/main). The `depends_on discourse` string is a non-fatal prepull-only warning, not the deploy. | **stale/PR-specific cc-ci OVERLAY test** mismatched to canonical-sweep context (not flake/timeout/recipe-deploy/warm-machinery) |
| mattermost-lts | DONE @00:05Z (`/tmp/redfix-mattermost-lts.log`) | install/upgrade/backup/custom PASS; **restore FAIL** `ci_marker does not exist` — **deterministic in isolation** (not a load race) | recipe `postgres` svc backup labels: backs up hot live PGDATA + dump but has **NO `backupbot.restore.post-hook`** to replay the dump → restore doesn't round-trip postgres. Contrast immich (passes): dump-only `backup.volumes.postgres.path: backup.sql` + `restore.post-hook: /pg_backup.sh restore`. | **genuine RECIPE defect** at latest → recipe PR (adopt immich-style dump+restore-post-hook) |
| mumble | DONE — **2× isolation GREEN** (`/tmp/redfix-mumble.log` + `/tmp/redfix-mumble2.log`) | **ALL tiers PASS** incl. handshake on BOTH runs; no orphans; canonical re-promoted green each time | handshake (TLS+ServerSync) not completing within ~60s retry under heavy concurrent sweep load; fine in isolation | **load/timing FLAKE** → harness stabilization (readiness gate / retry) |
| bluesky-pds | DONE @00:45Z (`/tmp/redfix-bluesky-pds.log` + live diag) | cold lifecycle GREEN; **WC5 promote 000** reproduces (warm /xrpc/_health last status 0). NOT a flake | caddy on-demand TLS (`ask http://app:3000/tls-check`) can't reach app: caddy resolves bare `app` to OTHER stacks' app endpoints on shared `proxy` net (getent app→only 10.10.0.X, never internal 10.0.3.3; proxy has drone/traefik/keycloak/ccci `app` aliases) → no cert → 000. Promote machinery correct (refused to write canonical). | **genuine routing/RECIPE defect** (cross-stack `app`-alias collision on shared proxy) → recipe PR: unique PDS service name/alias. NOT promote-machinery, NOT flake |
| gitea | DONE @00:14Z (`/tmp/redfix-gitea2.log` + live container logs) | cold lifecycle (incl fresh 3.5.3→3.6.0 upgrade) PASS; **warm advance crash-loops** | `LoadCommonSettings() [F] error saving JWT Secret … failed to save "/etc/gitea/app.ini": read-only file system` — gitea 3.6.0/1.24.2 tries to persist a JWT to the read-only app.ini docker-config mount on warm reattach (before DB migration; 3.5.3 data intact). Cold passes (fresh secrets, no rewrite). | **genuine RECIPE defect** (3.6.0 + read-only app.ini config mount on advance) → recipe PR: render app.ini into the writable config volume. (1st gitea run hit a nixenv "already deployed" leftover confound — fixed by undeploying to idle then re-running) |
| keycloak | DONE @01:05Z (code-verified; no run) | de-enrolled. `canonical_domain("keycloak")` == `WARM_DOMAINS["keycloak"]` == `warm-keycloak.ci.commoninternet.net` EXACTLY (canonical.py:42, warm.py:27,44). Live keycloak 200 /realms/master. | data-warm canonical domain uses same `warm-<r>` scheme as the live-warm OIDC provider → promote would collide with live shared SSO. No collision-free canonical namespace exists. | **HARNESS defect** (warm-domain namespace collision) → fix: collision-free `canonical_domain` for live-warm providers (`warm-canon-<r>`), then enroll keycloak |

### M1 results table (recipe → failure → isolation result → root cause → classification → fix approach)

| Recipe | Canon-sweep failure | Isolation result | Flake or genuine | Root cause | Class | Fix approach (M2) |
|---|---|---|---|---|---|---|
| discourse | "cold-deploy timeout / deploy FATA" | install/backup/restore/custom GREEN; **upgrade overlay RED** | **genuine (deterministic)** — but the canon root-cause was WRONG (no timeout, no deploy FATA) | cc-ci overlay `tests/discourse/test_upgrade.py` asserts head = official `discourse/discourse:3.5.3` + sidekiq dropped; that migration is in NO release tag and NOT in main (all use `bitnamilegacy/discourse:3.5.0`+sidekiq) | **stale/PR-specific cc-ci OVERLAY test** | make the overlay assert migration-faithfulness only when the head IS that migration (not vs a release tag), OR a recipe PR migrating off deprecated bitnamilegacy — settle in M2 (NOT a test-weakening) |
| mattermost-lts | `test_restore_returns_state` RED | install/upgrade/backup/custom GREEN; **restore RED** (`ci_marker does not exist`) | **genuine (deterministic in isolation)** — NOT the canon "loaded-node race" | recipe postgres backup labels back up hot PGDATA + a dump but have **no `backupbot.restore.post-hook`** to replay it; restore doesn't round-trip. immich (passes) uses dump-only path + `restore.post-hook` | **genuine RECIPE defect** | recipe PR: adopt immich-style postgres dump + `backupbot.restore.post-hook` replay |
| mumble | `test_handshake…` RED | **ALL tiers GREEN** in isolation (×N) incl. handshake | **FLAKE (load/timing)** | handshake (TLS+ServerSync) doesn't complete within the 60s retry under heavy concurrent sweep load; fine isolated; canonical written green today | **load/concurrency FLAKE** | harness stabilization: stronger readiness gate before the custom tier / longer-or-smarter handshake retry |
| bluesky-pds | warm promote `/xrpc/_health` → 000 | cold lifecycle GREEN; **warm promote 000 reproduces** | **genuine (deterministic)** — NOT a load/rate-limit flake | caddy on-demand TLS calls `http://app:3000/tls-check`; caddy resolves bare `app` to OTHER stacks' app endpoints on the shared `proxy` net (every stack aliases its main svc `app`), never bluesky's own internal app (10.0.3.3) → connection refused → no cert → 000 | **genuine ROUTING/RECIPE defect** (cross-stack `app`-alias collision) | recipe PR: give the PDS service a unique name/alias so caddy resolves only bluesky's app |
| gitea | 3.5.3→3.6.0 warm advance doesn't promote | cold (incl fresh upgrade) GREEN; **warm advance crash-loops** | **genuine (deterministic)** | gitea 3.6.0/1.24.2 saves a JWT secret to `/etc/gitea/app.ini` on warm reattach; app.ini is a **read-only docker-config mount** → `read-only file system` FATA at LoadCommonSettings (pre-migration; 3.5.3 data intact). Cold passes (fresh secrets, no rewrite) | **genuine RECIPE defect** | recipe PR: render app.ini into the writable `config:/etc/gitea` volume (entrypoint) instead of a read-only docker config |
| keycloak | de-enrolled (not tested) | code-verified (no run) | **genuine (structural)** | `canonical_domain("keycloak")` == `WARM_DOMAINS["keycloak"]` == `warm-keycloak.ci.commoninternet.net` EXACTLY → a data-warm canonical would collide with the live-warm OIDC provider | **HARNESS defect** (warm-domain namespace collision) | harness: collision-free `canonical_domain` for live-warm providers (`warm-canon-<r>`), then enroll keycloak (WARM_CANONICAL=True) |

### HOW the Adversary cold-verifies each classification (run ONE recipe at a time, no concurrent load)

Isolation invocation (per recipe `R` at latest tag `T`), from `/etc/cc-ci` on cc-ci:
`git -C ~/.abra/recipes/R checkout -f --quiet T && env -u REF -u CCCI_QUICK -u MODE -u VERSION RECIPE=R CCCI_SKIP_FETCH=1 cc-ci-run runner/run_recipe_ci.py`
Latest tags: discourse `0.8.1+3.5.0`, mattermost-lts `2.1.9+10.11.15`, mumble `1.0.0+v1.6.870-0`, bluesky-pds `0.3.0+v0.4.219`, gitea `3.6.0+1.24.2-rootless`.

- **discourse** — EXPECT install/backup/restore/custom pass, upgrade fail on `test_head_runs_official_image_not_bitnamilegacy` + `test_sidekiq_service_dropped_by_head`. Confirm the overlay mismatch statically: `git -C ~/.abra/recipes/discourse show 0.8.1+3.5.0:compose.yml | grep -A1 '  app:'` and `... show main:compose.yml` both = `bitnamilegacy/discourse:3.5.0`; `grep -c 'sidekiq:'` = 1 in both. So the test's `discourse/discourse:3.5.3`/no-sidekiq expectation exists nowhere upstream.
- **mattermost-lts** — EXPECT restore fail `relation "ci_marker" does not exist`. Confirm root cause statically: `git -C ~/.abra/recipes/mattermost-lts show 2.1.9+10.11.15:compose.yml | grep backupbot` shows pre-hook + `backup.path` but NO `restore.post-hook`; immich `git -C ~/.abra/recipes/immich show <latest>:compose.yml | grep backupbot` shows `restore.post-hook: /pg_backup.sh restore`.
- **mumble** — EXPECT all tiers green (run 2–3× to confirm reproducibly green isolated). Canonical written green: `cat /var/lib/ci-warm/mumble/canonical.json`.
- **bluesky-pds** — EXPECT cold green, WC5 promote `!! WC5 promote failed … warm-bluesky-pds … last status 0`. While the warm stack is up, confirm root cause: caddy logs `dial tcp 10.10.0.X:3000: connect: connection refused` for `app:3000/tls-check`; `docker exec <caddy> getent hosts app` returns proxy IPs (10.10.0.X), the app's real internal IP is 10.0.3.x; `docker network inspect proxy | grep _app` shows many stacks aliasing `app`. (Tear down the orphaned warm-bluesky-pds stack + volumes after.)
- **gitea** — REQUIRES idle canonical first: if warm-gitea is deployed, `docker stack rm warm-gitea_ci_commoninternet_net` (retains data+config volumes) so the advance reattaches from idle. EXPECT cold green, warm advance crash-loop with container log `LoadCommonSettings() [F] error saving JWT Secret … "/etc/gitea/app.ini": read-only file system`. Restore: leave warm-gitea undeployed (idle 3.5.3, volumes retained) — registry stays `3.5.3+1.24.2-rootless`.
- **keycloak** — no run. Code-verify: `canonical.canonical_domain('keycloak')` → `warm.stable_domain('keycloak')` → `warm-keycloak.ci.commoninternet.net`; `warm.WARM_DOMAINS['keycloak']` == same string (runner/harness/canonical.py:42-44, warm.py:27-29,44-48). Live keycloak 200 on `/realms/master`.

### Node state left clean
All isolation runs torn down; orphaned warm-bluesky-pds stack+volumes removed; warm-gitea restored to idle 3.5.3 (volumes retained, registry unchanged); only live warm-keycloak deployed (healthy). No `run_recipe_ci.py` processes.

## M1 — PASS @ 2026-06-18T01:18Z (REVIEW-redfix.md; all 6 classifications cold-verified CORRECT by Adversary's own isolation re-runs). No VETO. Cleared to M2.

## Phase: M2 — FIX + verify all six (IN PROGRESS)

Fix designs locked in BACKLOG-redfix.md. Recipe PRs (mattermost-lts/bluesky/gitea) on git.autonomic.zone
mirrors via the recipe mirror+PR flow, verified `!testme` (NEVER merge). Harness fixes (keycloak/mumble)
on a cc-ci branch, verified via the harness. discourse: overlay-scope decision. Node now free for my
deploys (Adversary done with M1).

### M2 fix tracker (updated 2026-06-18T05:53Z — ALL VERIFIED)

| Recipe | Class | Fix | PR/branch + ref | Status |
|---|---|---|---|---|
| mattermost-lts | recipe defect | pg_backup.sh + `backupbot.restore.post-hook` (immich pattern) | mirror PR #1 `ci/pg-restore` @4ca7f418 | **VERIFIED** — !testme run #901 ALL tiers green incl `test_restore_returns_state` |
| discourse | stale cc-ci overlay | recipe: bitnamilegacy->official discourse image migration + drop orphaned image-less sidekiq from compose.smtpauth.yml (F-redfix-1) | mirror PR #4 `discourse-official-image` @9ff5e19 | **VERIFIED** — own cold run `/tmp/redfix-discourse-m2verify.log` **level=5 of 5** (all tiers + lint R011 PASS); F-redfix-1 regression fixed |
| keycloak | harness defect | collision-free `canonical_domain` (`warm-canon-<r>` for WARM_DOMAINS recipes) + enroll | cc-ci branch `redfix-m2-harness` @61211db | **VERIFIED** — branch-checkout run promotes at warm-canon-keycloak; live warm-keycloak 200 throughout |
| mumble | load/timing flake | harness: handshake readiness budget 60s->180s | cc-ci branch `redfix-m2-harness` @07fc6d4 | **VERIFIED** — branch-checkout run all tiers green incl handshake; budget active+non-regressing |
| gitea | recipe defect | app.ini->staging `/etc/gitea/app.ini.init` + docker-setup seed-on-EMPTY + DOCKER_SETUP_SH_VERSION v3 | mirror PR #2 `ci/app-ini-writable` @a0f2db8 | **VERIFIED** (direct chaos-deploy; promote merge-gated — see below) |
| bluesky-pds | recipe defect (routing) | caddy `{$APP_HOST}=${STACK_NAME}_app` (operator: NO rename) + CADDYFILE_VERSION v2 | mirror PR #4 `ci/warm-routing-alias` @4987ba9 | **VERIFIED** (direct chaos-deploy; promote merge-gated — see below) |

cc-ci-side change verification: run from a checkout of `redfix-m2-harness` (CCCI_REPO=<checkout>);
never touches /etc/cc-ci main. `redfix-m2-harness` is now mumble+keycloak ONLY (bluesky needs no
cc-ci change with the ${STACK_NAME}_app approach; the rename's exec-ref commit b96b8a4 was dropped).

## Gate: M2 — RE-CLAIMED, awaiting Adversary (2026-06-18T06:55Z; orig claim 05:53Z)

**Re-claim delta (addresses Adversary M2 FAIL @06:42Z — finding F-redfix-1).** The first M2 verdict was
FAIL on discourse ONLY (other 5 PASS, do-not-redo). F-redfix-1: the official-image migration dropped
`sidekiq` from compose.yml but left a dangling image-less `sidekiq:` block in `compose.smtpauth.yml` →
L5 lint R011 fail (run level=4) + broken SMTP-auth deploy. **FIXED** in PR #4 `discourse-official-image`
@**9ff5e19** (force-pushed onto @53ba0910): dropped the orphaned `sidekiq:` block; the `app:` override
already carries `DISCOURSE_SMTP_PASSWORD_FILE` + `smtp_password` secret (sidekiq is internal to the
official image), so no SMTP coverage lost. `grep sidekiq compose*.yml` = 0.
**VERIFIED two ways:** (1) the Adversary's exact lint.py repro flow at 9ff5e19 → **R011 ✅**; (2) my own
full cold run `/tmp/redfix-discourse-m2verify.log` → `RUN SUMMARY ... level=5 of 5`, all tiers pass
(install/upgrade/backup/restore/custom), `lint rung: pass`. Node clean: no discourse stack, NO discourse
canonical (untagged migrated head correctly does not promote — should_promote tagged-gate), recipe reset
to published tag 0.8.1+3.5.0. The other 5 fixes are unchanged since their Adversary PASS (keycloak,
mumble, gitea, bluesky-pds, mattermost-lts) — no re-run needed.

Adversary cold-verify for discourse: clone discourse @9ff5e19, run `RECIPE=discourse CCCI_SKIP_FETCH=1
… run_recipe_ci.py` → EXPECT level=5 of 5 (lint R011 ✅, all tiers pass, both upgrade-overlay tests
`test_head_runs_official_image_not_bitnamilegacy` + `test_sidekiq_service_dropped_by_head` pass); OR the
lint-only repro in F-redfix-1 → R011 ✅. `grep -c sidekiq ~/.abra/recipes/discourse/compose*.yml` @9ff5e19 = 0.

---
## Gate: M2 — original claim (2026-06-18T05:53Z)

**WHAT (M2 DoD).** All six canon-sweep failures FIXED — each via a recipe PR or a harness improvement —
and verified green. No recipe left as a standing exception. Nothing merged (operator merges). Per recipe:

- **mattermost-lts** (recipe PR #1) — added `pg_backup.sh` + postgres `backupbot.restore.post-hook` so
  the logical dump round-trips on restore.
- **discourse** (recipe PR #4) — migrated the head off deprecated `bitnamilegacy` to the official
  `discourse/discourse` image so the stale PR-faithfulness overlay (`test_head_runs_official_image…`,
  `test_sidekiq_service_dropped…`) passes on the migrated head (NOT a test-weakening).
- **keycloak** (harness branch) — `canonical_domain` returns a collision-free `warm-canon-<r>` for
  recipes in `warm.WARM_DOMAINS` (live-warm OIDC providers); keycloak enrolled (WARM_CANONICAL=True).
- **mumble** (harness branch) — handshake readiness budget widened 60s->180s (load-flake stabilization).
- **gitea** (recipe PR #2) — app.ini is now seeded into the WRITABLE `/etc/gitea` volume by
  docker-setup (`if [ ! -s /etc/gitea/app.ini ]`, seed-on-EMPTY) from the read-only staging config
  `app.ini.init`; `DOCKER_SETUP_SH_VERSION` v1->v3 forces the new docker-setup to re-mount. Gitea
  1.24.2 can then persist its JWT secret (the M1 read-only-app.ini crash is gone).
- **bluesky-pds** (recipe PR #4) — caddy resolves its OWN app via the fully-qualified swarm name
  `${STACK_NAME}_app` (caddy `{$APP_HOST}` env, set in the caddy service) instead of bare `app`, which
  collided with other stacks' `app` aliases on the shared `proxy` net. CADDYFILE_VERSION v1->v2.

**HOW + EXPECTED + WHERE (Adversary cold-verify, one recipe at a time, no concurrent load):**

- **mattermost-lts** — read-only artifact: `/var/lib/cc-ci-runs/901/` on cc-ci — all tiers pass,
  `junit/restore__cc-ci__test_restore.xml` testsuite failures=0, `test_restore_returns_state` pass.
  OR re-run !testme on PR #1 @4ca7f418. EXPECT restore green.
- **discourse** — !testme on PR #4 @53ba0910 (run #849 green) OR run from a checkout of the migrated
  head: EXPECT install/backup/restore/custom + upgrade overlay all pass (head now official image).
- **keycloak** — from a `redfix-m2-harness` @61211db checkout (CCCI_REPO=<checkout>), run
  `RECIPE=keycloak CCCI_SKIP_FETCH=1 ... run_recipe_ci.py`. EXPECT all cold tiers pass + WC5 promote
  succeeds at domain `warm-canon-keycloak.ci.commoninternet.net` (NOT warm-keycloak); live
  `warm-keycloak.ci.commoninternet.net/realms/master` stays 200 throughout. Code: `canonical.py`
  canonical_domain returns warm-canon-<r> for r in warm.WARM_DOMAINS.
- **mumble** — from `redfix-m2-harness` @07fc6d4 checkout, run `RECIPE=mumble CCCI_SKIP_FETCH=1 …`.
  EXPECT all 5 tiers green incl `custom/test_protocol_handshake.py::test_handshake_completes_with_
  channel_presence`; handshake budget = 36 attempts / 180s (was 60s). (Load-flake is not
  deterministically reproducible; this verifies the stabilization is applied, sound, non-weakening.)
- **gitea** (recipe PR #2 @a0f2db8 on mirror branch `ci/app-ini-writable`) — DIRECT chaos-deploy proof
  (the harness WC5 promote is merge-gated, see NOTE). With the idle 3.5.3 canonical present:
  `cd ~/.abra/recipes/gitea && git checkout -f a0f2db8` then chaos-deploy onto the retained canonical
  volumes (0-byte app.ini = genuine pre-fix 3.5.3 state):
  `abra app deploy warm-gitea.ci.commoninternet.net -C -o -n`. EXPECT: service 1/1; the config volume's
  `app.ini` seeded 0->~1862 bytes (`INSTALL_LOCK = true`); `/api/v1/version` -> 200 {"version":"1.24.2"}
  and `/api/healthz` -> 200 (curl inside the app container); retained 3.5.3 data adopted (data dirs
  dated 2026-06-17T08:39); ZERO `read-only file system` crashes in `docker service logs` (M1 crashed
  here). Evidence: `/tmp/redfix-gitea-m2-directproof.log` on cc-ci. Teardown: `abra app undeploy … -n`,
  truncate the volume app.ini to 0 (restore pre-fix state). canonical.json stays 3.5.3 idle e6a1cc79.
- **bluesky-pds** (recipe PR #4 @4987ba9 on mirror branch `ci/warm-routing-alias`) — DIRECT chaos-deploy
  proof (warm-promote is the only failing path; merge-gated). `git checkout -f 4987ba9`; generate
  secrets (`abra app secret generate warm-bluesky-pds.ci.commoninternet.net --all -m -C -o -n`) + insert
  a PLC rotation key (tests/bluesky-pds/install_steps.sh logic: 32-byte hex into pds_plc_rotation_key
  v1); **re-checkout 4987ba9 AFTER secret ops** (abra secret insert force-fetches+reverts the checkout);
  `abra app deploy warm-bluesky-pds.ci.commoninternet.net -C -o -n` (EXPECT `caddyfile: v1 -> v2`,
  NEW DEPLOYMENT 4987ba9). EXPECT: app+caddy 1/1; inside caddy `getent hosts
  warm-bluesky-pds_ci_commoninternet_net_app` -> a 10.0.x.x INTERNAL ip (own stack) while
  `getent hosts app` -> a 10.10.x.x proxy ip (foreign, the M1 collision); caddy log "certificate
  obtained successfully" with 0 "connection refused"; external `curl https://warm-bluesky-pds.ci.
  commoninternet.net/xrpc/_health` -> **200** {"version":"0.4.219"} (M1 was 000). Evidence:
  `/tmp/redfix-bluesky-m2-directproof.log`. Teardown: undeploy + remove volumes (caddy_data, pds_data)
  + secrets (no canonical, matching M1).

**NOTE — gitea & bluesky end-to-end canonical-promote is OPERATOR-MERGE-GATED (not a shrug).** The
harness WC5 promote does a recipe_checkout(published-tag)+non-chaos deploy, and BOTH run_recipe_ci.py:373
AND abra force-fetch `refs/tags/*` from upstream (abra.py:135 documents this), so any local move of the
release tag to the fix commit is reverted to the PUBLISHED commit. The published 3.6.0 / 0.3.0 tags do
NOT yet carry the fix (PR not merged — operator merges, per phase guardrail), so pre-merge the promote
necessarily deploys the unfixed published release. Confirmed empirically: a full gitea harness run's WC5
promote deployed 357926f and crash-looped exactly like M1. The DIRECT chaos-deploy (chaos = deploy the
working-tree checkout = the PR fix) is therefore the MAXIMAL + faithful pre-merge proof — it reproduces
the EXACT M1 failing scenario (gitea: the retained canonical volumes; bluesky: warm-bluesky-pds on the
shared proxy) and shows the fix resolves it. End-to-end canonical advance follows automatically once the
operator merges PR #2 / #4 and the release tag carries the fix. This is NOT a standing exception — the
defect is fixed + proven; only the registry-advance awaits the operator's merge (the phase's own
"nothing merged" constraint).

**WHERE (refs).** Recipe PRs on `git.autonomic.zone/recipe-maintainers/<recipe>`: mattermost-lts
`ci/pg-restore`@4ca7f418, discourse `discourse-official-image`@53ba0910, gitea `ci/app-ini-writable`
@a0f2db8, bluesky-pds `ci/warm-routing-alias`@4987ba9. cc-ci harness branch
`redfix-m2-harness`@07fc6d4 (keycloak 61211db + mumble 07fc6d4). Reasoning/dead-ends in
JOURNAL-redfix.md. Node left clean (only infra + live warm-keycloak 200; gitea idle 3.5.3 volumes
retained, canonical e6a1cc79 unchanged; no bluesky/test stacks/volumes/secrets; no run procs).

## Gate: M1 — PASS (above).

**WHAT (M1 DoD).** All six canon-sweep failures investigated in ISOLATION (one recipe at a time, no
concurrent sweep load), root-caused with first-hand evidence, and classified (flake vs genuine; recipe
vs test vs warm-machinery vs load) — see the **M1 results table** + **HOW the Adversary cold-verifies**
sections above. Summary: discourse = stale cc-ci overlay test (canon timeout/FATA root-cause was
wrong); mattermost-lts = genuine recipe defect (no `backupbot.restore.post-hook`); mumble = load/timing
FLAKE (2× isolation green); bluesky-pds = genuine routing defect (caddy↔app `app`-alias collision on
shared proxy); gitea = genuine recipe defect (read-only app.ini config mount + 3.6.0 JWT save);
keycloak = harness warm-domain namespace collision. NO "probably a flake" — every classification has
an isolation re-run or code proof.

**HOW + EXPECTED + WHERE.** Per-recipe cold-verify commands, expected outputs, and evidence paths are
in the two sections above ("M1 results table" and "HOW the Adversary cold-verifies each classification").
Evidence logs on cc-ci: `/tmp/redfix-{discourse,mattermost-lts,mumble,mumble2,bluesky-pds,gitea2}.log`.
Reasoning/dead-ends in JOURNAL-redfix.md. Node left clean (see "Node state left clean" above).

## Untracked file `main.go` — NOT produced by this phase (operator: unexplained)

**WHAT.** An untracked `main.go` (281 bytes, a Go "Hello, World!" `net/http` server on `:8080`) sits in the
repo root. It is not in any commit on any ref, is not gitignored, and is unrelated to every redfix DoD item.
No `go.mod` accompanies it. I did not create it and have not committed, modified, or deleted it.

**WHERE.** `/srv/cc-ci-orch/cc-ci/main.go` **and** `/srv/cc-ci/cc-ci/main.go` — present in **both** clones,
identical size, identical mtime `2026-07-09 00:01`, i.e. written by something outside git (a git operation
cannot leave the same untracked file in two clones).

**HOW to verify.** `git log --all --oneline -- main.go` → empty (in no commit).
`ls -la /srv/cc-ci-orch/cc-ci/main.go /srv/cc-ci/cc-ci/main.go` → both 281 bytes, mtime `Jul 9 00:01`.
`git status --short` → `?? main.go`.

**EXPECTED.** Left in place, untracked. Per guardrails I do not delete or commit files I did not create;
flagging for the operator rather than acting. It affects no gate, no DoD item, and no verdict.

## Blocked

(none)