# STATUS — phase `redfix` Phase SSOT: `/srv/cc-ci/cc-ci-plan/plan-phase-redfix-canon-sweep-failures.md` Mission: investigate every canon-sweep failure (discourse, mattermost-lts, mumble, bluesky-pds, gitea, keycloak) → isolate → root-cause → classify (flake vs genuine; recipe vs test vs warm-machinery vs load) → FIX each (recipe PR or harness improvement) → verify green. No standing exceptions. Nothing merged. ## DONE — 2026-07-09T00:18Z Phase `redfix` COMPLETE. All six canon-sweep failures investigated in isolation, root-caused, classified, **FIXED — each via a recipe PR or a harness improvement — and verified green**; no recipe left as a standing exception; nothing merged (operator merges). Every gate has a fresh Adversary PASS in REVIEW-redfix.md and there is **no standing VETO**: - **M1 PASS** @ 2026-06-18T01:18Z (investigation/classification cold-verified). - **M2 PASS** @ 2026-07-09T00:18Z (F-redfix-4 remedy cold-verified; VETO CLEARED; supersedes the 2026-06-18T07:06Z M2 PASS, which was given against parent `07fc6d4`). Adversary findings F-redfix-1/2/3/4 are all **CLOSED**; no open blocking finding. ## ⚠ OPERATOR ACTION REQUIRED — B-redfix-8 (HIGH, open, outside DoD) Phase DoD is met, but **one HIGH item is open and only an operator can close it.** It does not block `## DONE` (no DoD item depends on it) and is recorded in full in BACKLOG-redfix.md. **WHAT.** The live Gitea bot password (`GITEA_PASSWORD` in `/srv/cc-ci/.testenv`) was committed in `14c7dee` (`machine-docs/BACKLOG-redfix.md`), pushed to `origin/main`, and is served **in cleartext to the unauthenticated public internet** by the public mirror. It is push-capable to `recipe-maintainers/*`. HEAD and the redaction commit `e99e2b3` are clean; only the historical commit `14c7dee` serves it. Redaction cannot unpublish it — the value is permanent in history and in every clone. **WHERE.** `git.autonomic.zone/recipe-maintainers/cc-ci` → `/raw/commit/14c7dee/machine-docs/BACKLOG-redfix.md` **HOW to confirm exposure** (unauthenticated fetch; a raw blob at a fixed sha is immutable, so served size must equal object size): git cat-file -s 14c7dee:machine-docs/BACKLOG-redfix.md # → 33408 An unauthenticated fetch of the raw URL returning **HTTP 200 / 33408 bytes** is the leak, not an error page. **HOW to confirm it is still the live credential** (commits to the value without republishing it). Run this **on the orchestrator** — `/srv/cc-ci/.testenv` lives there, **not** on `cc-ci`: python3 -c 'import hashlib,re;v=re.search(r"^GITEA_PASSWORD=(.*)$",open("/srv/cc-ci/.testenv").read(),re.M).group(1).strip().strip(chr(34)).strip(chr(39));print(hashlib.sha256(v.encode()).hexdigest()[:16])' **EXPECTED.** Prints `3fcea78925015fc9` while still unrotated. **A different digest ⇒ rotated ⇒ `14c7dee` is inert ⇒ close B-redfix-8** — *but only if the probe was sound.* Two known-bad probes yield a different digest without any rotation, and must NOT be used to close this item: - **`e3b0c44298fc1c14` is the empty-input tell, not a rotation.** Running the command over `ssh cc-ci` fails to open `.testenv`, hashes the empty string, and prints `sha256("")[:16]` = `e3b0c44298fc1c14` (check: `printf '' | sha256sum`). Fix the probe; do not close B-redfix-8. - **Do not conflate the three digests.** `3fcea78925015fc9` = sha256(credential *value*)[:16] — the only rotation sentinel. `1994fd8d…` = sha256(the whole served *blob*) — immutable, changes never. `e3b0c44298fc1c14` = sha256(empty) — a broken probe. Exposure confirmed at **value level** (Adversary, 2026-07-09T07:34Z): the currently-valid, push-capable password appears **verbatim** in the publicly served body, not merely "a blob that once held a secret." **REMEDY (operator).** Rotate the Gitea bot password, then update `/srv/cc-ci/.testenv`. The credential is a Class-A1 external infra input (plan §4.4) — the Builder must not rotate or invent it. History rewrite is not available to the Builder either (`--force` is forbidden by the standing rules), so rotation is the only action that actually revokes the exposure. **Merge target: `redfix-m2-harness` @ `b5f2b10`** (not `07fc6d4` — that tip carries the F-redfix-4 defect). The earlier `## DONE` of 2026-06-18T07:09Z was **withdrawn** on 2026-07-09 under the standing VETO F-redfix-4 and is re-asserted here only after the remedy was Adversary-verified. Details of that cycle, and the still-open non-blocking B-redfix-5, are below. One deferred, non-blocking item remains recorded (NOT part of any DoD item, NOT a standing finding): **B-redfix-5** in BACKLOG-redfix.md — `warm_reconcile.py`'s rollback `warmsnap.restore()` sits outside the upgrade's `try/except`. Pre-dates the enrollment (present at `07fc6d4`); F-redfix-4 supplied the only *reachable* trigger and that is now closed. Adversary concurred it is not a VETO. --- ### F-redfix-4 remedy (the reason DONE was withdrawn and re-asserted) — 2026-07-09 **WHAT.** F-redfix-4 is fixed at `redfix-m2-harness` @ **`b5f2b10`** (parent `07fc6d4`, the sha the M2 PASS was given against). Warm state is now keyed by a **stack namespace**, not a bare recipe: `canonical.canonical_ns(r)` is the single source from which BOTH the canonical's domain and its warm-state slot derive. A live-warm provider (`r in warm.WARM_DOMAINS`) gets ns `canon-` → domain `warm-canon-keycloak.ci.commoninternet.net` (**unchanged**) + slot `/var/lib/ci-warm/canon-keycloak/`. Every other recipe keeps ns `` (the invariant is structural — `r not in warm.WARM_DOMAINS` — not a property of any particular count). `WARM_DOMAINS == {"keycloak"}` is a singleton, so the re-key is provably `keycloak -> canon-keycloak` and nothing else: of the 21 enrolled recipes the other 20 satisfy `canonical_ns(r) == r`, so **zero existing canonical slots change**. On live disk that is 17 seeded canonicals (those carrying a `canonical.json`), all of which remain in `prune_stale`'s `keep` set. Addresses all four consequences: (1)+(2) slots disjoint, neither deployment can replace the other's known-good; (3) the outage race is removed with the shared slot; (4) `prune_stale`'s reconciler-dir invariant is now structural — `/` never gains a `canonical.json`. Also: `warmsnap._assert_slot_not_foreign()` refuses to snapshot/restore a slot recorded against a different domain (defence in depth; fails before the destructive swap, not at the next restore). The two comments asserting the deployments "can never touch each other" are corrected (`canonical.py`, `tests/keycloak/recipe_meta.py`). `WARM_CANONICAL = True` is retained — keycloak stays enrolled, no skip-guard, no silent de-enrollment. **MIGRATION: none required.** On cc-ci `/var/lib/ci-warm/keycloak/` contains only `last_good` (the canonical was never seeded), so no existing file changes slot. Verify: `ls -A /var/lib/ci-warm/keycloak/` → `last_good` only. **WHERE.** cc-ci repo, branch `redfix-m2-harness` @ `b5f2b10`. Files: `runner/harness/warmsnap.py`, `runner/harness/canonical.py`, `runner/warm_reconcile.py`, `runner/run_recipe_ci.py`, `tests/keycloak/recipe_meta.py`, `tests/unit/test_warmsnap.py`, `tests/unit/test_canonical.py`. **HOW to verify (1) — unit suite, cold clone, no docker needed.** git clone -q --branch redfix-m2-harness /tmp/v && cd /tmp/v /nix/store/x188l04r3gfkh18gy1dpf05fv3kkrgs7-python3-3.12.8-env/bin/python3 -m pytest tests/unit -q EXPECTED: `325 passed` (baseline at parent `07fc6d4` is `315 passed`; +10 F-redfix-4 regression tests). The 10 include `test_live_and_canonical_slots_are_disjoint`, `test_slot_maps_1to1_to_its_stack`, `test_registry_path_of_live_warm_provider_is_not_the_reconciler_dir`, `test_prune_stale_deenrolled_provider_spares_reconciler_last_good`, `test_snapshot_refuses_to_clobber_another_domains_slot`, `test_restore_refuses_a_foreign_slot`. **HOW to verify (2) — the VETO's clearing condition, on the real node.** Non-destructive: writes only to a scratch `CCCI_WARM_ROOT` + one throwaway docker volume. Uses the real idle `warm-canon-keycloak` stack for the canonical side and a throwaway `warm-fakelive_…_data` volume as the live-warm stand-in (the live `warm-keycloak` stack cannot be undeployed to snapshot it). ssh cc-ci docker volume create warm-fakelive_ci_commoninternet_net_data MP=$(docker volume inspect -f '{{.Mountpoint}}' warm-fakelive_ci_commoninternet_net_data); echo x > $MP/live.txt git clone -q --branch redfix-m2-harness /tmp/v && mkdir -p /tmp/vw/keycloak echo '10.7.1+26.6.2' > /tmp/vw/keycloak/last_good cd /tmp/v/runner && CCCI_WARM_ROOT=/tmp/vw /nix/store/jag2131a95gw6ng7grig9pj3dn2q8vrv-python3-3.12.8-env/bin/python3 - <<'PY' import sys; sys.path.insert(0, ".") from harness import warmsnap as ws, canonical as c LIVE, CANON = ws.live_slot("keycloak"), c.canonical_slot("keycloak") CANON_DOM, LIVE_DOM = c.canonical_domain("keycloak"), "warm-fakelive.ci.commoninternet.net" print(ws.snap_dir(LIVE), ws.snap_dir(CANON), c.registry_path("keycloak"), sep="\n") ws.snapshot(CANON, CANON_DOM, version="canon-known-good") ws.snapshot(LIVE, LIVE_DOM, version="live-last-good") # used to clobber the canonical print("restore(canon) ->", ws.restore(CANON, CANON_DOM)["volumes"]) print("restore(live) ->", ws.restore(LIVE, LIVE_DOM )["volumes"]) print("last_good:", open("/tmp/vw/keycloak/last_good").read().strip()) try: ws.snapshot(CANON, LIVE_DOM); print("FAIL: foreign snapshot allowed") except ws.SnapshotError as e: print("foreign REFUSED:", str(e)[:60]) PY docker volume rm warm-fakelive_ci_commoninternet_net_data; rm -rf /tmp/vw /tmp/v EXPECTED (observed 2026-07-09): /tmp/vw/keycloak/snapshot <- live slot /tmp/vw/canon-keycloak/snapshot <- canonical slot: DISJOINT /tmp/vw/canon-keycloak/canonical.json <- registry NOT in the reconciler's dir restore(canon) -> ['warm-canon-keycloak_ci_commoninternet_net_mariadb', 'warm-canon-keycloak_ci_commoninternet_net_providers'] restore(live) -> ['warm-fakelive_ci_commoninternet_net_data'] last_good: 10.7.1+26.6.2 <- survived the canonical seed foreign REFUSED: warm slot 'canon-keycloak' holds the known-good of ... Each `restore()` returns its OWN stack's volumes (the clearing condition). Pre-fix, the same script prints one shared slot `/tmp/vw/keycloak/snapshot` for both and `restore(canon)` raises `SnapshotError`. **Node integrity after my own run of the above (2026-07-09).** Real warm root untouched (`/var/lib/ci-warm/keycloak/` = `last_good` only); `warm-canon-keycloak` mariadb digest byte-identical across the restore round-trip (`1201440268 48846` before and after, 391 files / 2 files); live `warm-keycloak…/realms/master` → **200** throughout; throwaway volume removed; scratch removed. **Scope.** keycloak only. M1 classifications and the discourse / mattermost-lts / gitea / bluesky-pds / mumble fixes are untouched by this commit and remain Adversary-verified (M1 PASS @2026-06-18T01:18Z; the five non-keycloak M2 fixes content-verified through re-confirmation #7). **Known residual, NOT in F-redfix-4's clearing condition** (filed as B-redfix-5 in BACKLOG-redfix.md, not blocking): `warm_reconcile.py`'s rollback `warmsnap.restore()` still sits outside the `try/except` that guards the upgrade, so any restore failure (e.g. a corrupt/absent snapshot) leaves live keycloak undeployed after `abra.undeploy()`. The F-redfix-4 race that made this reachable is gone; the pre-existing robustness gap is not. Recorded, not silently dropped. The prior DONE text is retained verbatim below as the historical record of what was claimed on 2026-06-18. --- ## (HISTORICAL — withdrawn 2026-07-09 under VETO F-redfix-4; superseded by the DONE above) DONE — 2026-06-18T07:09Z Phase `redfix` COMPLETE. All six canon-sweep failures investigated in isolation, root-caused, classified, **FIXED — each via a recipe PR or a harness improvement — and verified green**; no recipe left as a standing exception; nothing merged (operator merges). Both gates have a fresh Adversary PASS in REVIEW-redfix.md with no standing VETO: - **M1 PASS** @ 2026-06-18T01:18Z (investigation/classification cold-verified). - **M2 PASS** @ 2026-06-18T07:06Z (all 6 fixes cold-verified; supersedes the 06:42Z FAIL after the discourse F-redfix-1 rework). Fixes (per recipe): mattermost-lts recipe PR #1 (pg_backup.sh + restore.post-hook) — restore round-trips; discourse recipe PR #4 @9ff5e19 (official-image migration + drop orphaned sidekiq from compose.smtpauth.yml) — level=5, lint R011 ✅; keycloak harness (collision-free `warm-canon-` + enroll) — promotes without touching live SSO; mumble harness (handshake budget 60→180s) — flake stabilized, non-weakening; gitea recipe PR #2 @a0f2db8 (app.ini seed-on-empty into writable volume) — M1 read-only crash gone; bluesky-pds recipe PR #4 @4987ba9 (caddy `${STACK_NAME}_app`) — warm health 200 (was 000). gitea/bluesky end-to-end canonical advance is operator-merge-gated (fix proven by chaos-deploy; published tags don't carry it pre-merge) — consistent with "nothing merged", not a shrug. ### Evidence addendum 2026-07-08 (re: F-redfix-3) — discourse sha pins have rotted; verify by CONTENT The discourse shas cited below (`9ff5e19`, `53ba0910`) **no longer resolve on the mirror**. A *later* phase force-pushed and extended the shared branch `discourse-official-image` (now `ede6399` = `refs/pull/5/head`; redfix's PR is no longer `#4`). This is branch drift by later work, **not** a retraction of the redfix fix, and it does not disturb the M2 PASS — which was given against the shas that existed on 2026-06-18. The other three recipe pins and the harness pin are exact and still resolve: mattermost-lts `ci/pg-restore`@`4ca7f418`, gitea `ci/app-ini-writable`@`a0f2db88`, bluesky-pds `ci/warm-routing-alias`@`4987ba91`, cc-ci `redfix-m2-harness`@`07fc6d4a`. Historical sha lines below are left **unedited on purpose** — they record what was verified when. Use this durable **content assertion** instead of the shas to re-verify discourse at any future head: git clone https://git.autonomic.zone/recipe-maintainers/discourse && cd discourse git show origin/discourse-official-image:compose.yml | grep -m1 'image:.*discourse' git show origin/discourse-official-image:compose.smtpauth.yml | grep -c sidekiq EXPECTED: `image: discourse/discourse:3.5.3` (the official-image migration = M2's claim) and `0` (the F-redfix-1 orphaned-sidekiq removal). Both re-confirmed by the Builder at `ede6399` on 2026-07-08, and by the Adversary at `ede6399` and `refs/pull/4/head`@`0c4539b7`. (Caveat when reproducing absence of a sha: a `--filter=blob:none` clone and `git fetch origin ` both yield false "absent" signals — test reachability from all `refs/heads/*` + `refs/pull/*/head` instead.) --- ## Phase: M1 — investigate + isolate + classify (IN PROGRESS) Bootstrapped 2026-06-17T23:20Z. cc-ci healthy, no run in flight, next scheduled sweep 2026-06-21 (3-day clear window). Disk `/` 38G free (75% used). ### Isolation harness (how I reproduce each failure ALONE) Each canon-sweep per-recipe run is `runner/nightly_sweep.run_on_tag(recipe, latest)`: `abra.recipe_checkout(recipe, )` then `run_recipe_ci.py` with `RECIPE= CCCI_SKIP_FETCH=1` and REF/QUICK/MODE/VERSION unset (cold, full, head==tag). Isolation = run ONE recipe at a time with NO concurrent sweep load on the single node (the loaded node is the known flake source per phase plan §2.1). Runs execute on cc-ci from `/etc/cc-ci`. ### Starting canonical state (cc-ci `/var/lib/ci-warm//canonical.json`, read 2026-06-17T23:19Z) | Recipe | Canonical now | Note | |---|---|---| | discourse | (none) | no canonical dir | | mattermost-lts | (none) | no canonical dir | | mumble | `1.0.0+v1.6.870-0` @ 20260617T180501Z | **canonical PRESENT, written TODAY** — flake signal | | bluesky-pds | (none) | no canonical dir | | gitea | `3.5.3+1.24.2-rootless` @ 20260617T083930Z | 3.6.0 advance not promoted | | keycloak | (none) | de-enrolled (WARM_CANONICAL off) | ### M1 investigation tracker | Recipe | Isolation run | Result | Root cause | Classification | |---|---|---|---|---| | discourse | DONE @23:40Z (`/tmp/redfix-discourse.log` on cc-ci) | install/backup/restore/custom PASS; **upgrade overlay FAIL**. Deploys+serves fine — NOT a timeout/FATA. | cc-ci overlay `tests/discourse/test_upgrade.py` asserts head runs official `discourse/discourse:3.5.3` + drops sidekiq; latest tag `0.8.1+3.5.0` AND main both still `bitnamilegacy/discourse:3.5.0`+sidekiq (migration exists in no release/main). The `depends_on discourse` string is a non-fatal prepull-only warning, not the deploy. | **stale/PR-specific cc-ci OVERLAY test** mismatched to canonical-sweep context (not flake/timeout/recipe-deploy/warm-machinery) | | mattermost-lts | DONE @00:05Z (`/tmp/redfix-mattermost-lts.log`) | install/upgrade/backup/custom PASS; **restore FAIL** `ci_marker does not exist` — **deterministic in isolation** (not a load race) | recipe `postgres` svc backup labels: backs up hot live PGDATA + dump but has **NO `backupbot.restore.post-hook`** to replay the dump → restore doesn't round-trip postgres. Contrast immich (passes): dump-only `backup.volumes.postgres.path: backup.sql` + `restore.post-hook: /pg_backup.sh restore`. | **genuine RECIPE defect** at latest → recipe PR (adopt immich-style dump+restore-post-hook) | | mumble | DONE — **2× isolation GREEN** (`/tmp/redfix-mumble.log` + `/tmp/redfix-mumble2.log`) | **ALL tiers PASS** incl. handshake on BOTH runs; no orphans; canonical re-promoted green each time | handshake (TLS+ServerSync) not completing within ~60s retry under heavy concurrent sweep load; fine in isolation | **load/timing FLAKE** → harness stabilization (readiness gate / retry) | | bluesky-pds | DONE @00:45Z (`/tmp/redfix-bluesky-pds.log` + live diag) | cold lifecycle GREEN; **WC5 promote 000** reproduces (warm /xrpc/_health last status 0). NOT a flake | caddy on-demand TLS (`ask http://app:3000/tls-check`) can't reach app: caddy resolves bare `app` to OTHER stacks' app endpoints on shared `proxy` net (getent app→only 10.10.0.X, never internal 10.0.3.3; proxy has drone/traefik/keycloak/ccci `app` aliases) → no cert → 000. Promote machinery correct (refused to write canonical). | **genuine routing/RECIPE defect** (cross-stack `app`-alias collision on shared proxy) → recipe PR: unique PDS service name/alias. NOT promote-machinery, NOT flake | | gitea | DONE @00:14Z (`/tmp/redfix-gitea2.log` + live container logs) | cold lifecycle (incl fresh 3.5.3→3.6.0 upgrade) PASS; **warm advance crash-loops** | `LoadCommonSettings() [F] error saving JWT Secret … failed to save "/etc/gitea/app.ini": read-only file system` — gitea 3.6.0/1.24.2 tries to persist a JWT to the read-only app.ini docker-config mount on warm reattach (before DB migration; 3.5.3 data intact). Cold passes (fresh secrets, no rewrite). | **genuine RECIPE defect** (3.6.0 + read-only app.ini config mount on advance) → recipe PR: render app.ini into the writable config volume. (1st gitea run hit a nixenv "already deployed" leftover confound — fixed by undeploying to idle then re-running) | | keycloak | DONE @01:05Z (code-verified; no run) | de-enrolled. `canonical_domain("keycloak")` == `WARM_DOMAINS["keycloak"]` == `warm-keycloak.ci.commoninternet.net` EXACTLY (canonical.py:42, warm.py:27,44). Live keycloak 200 /realms/master. | data-warm canonical domain uses same `warm-` scheme as the live-warm OIDC provider → promote would collide with live shared SSO. No collision-free canonical namespace exists. | **HARNESS defect** (warm-domain namespace collision) → fix: collision-free `canonical_domain` for live-warm providers (`warm-canon-`), then enroll keycloak | ### M1 results table (recipe → failure → isolation result → root cause → classification → fix approach) | Recipe | Canon-sweep failure | Isolation result | Flake or genuine | Root cause | Class | Fix approach (M2) | |---|---|---|---|---|---|---| | discourse | "cold-deploy timeout / deploy FATA" | install/backup/restore/custom GREEN; **upgrade overlay RED** | **genuine (deterministic)** — but the canon root-cause was WRONG (no timeout, no deploy FATA) | cc-ci overlay `tests/discourse/test_upgrade.py` asserts head = official `discourse/discourse:3.5.3` + sidekiq dropped; that migration is in NO release tag and NOT in main (all use `bitnamilegacy/discourse:3.5.0`+sidekiq) | **stale/PR-specific cc-ci OVERLAY test** | make the overlay assert migration-faithfulness only when the head IS that migration (not vs a release tag), OR a recipe PR migrating off deprecated bitnamilegacy — settle in M2 (NOT a test-weakening) | | mattermost-lts | `test_restore_returns_state` RED | install/upgrade/backup/custom GREEN; **restore RED** (`ci_marker does not exist`) | **genuine (deterministic in isolation)** — NOT the canon "loaded-node race" | recipe postgres backup labels back up hot PGDATA + a dump but have **no `backupbot.restore.post-hook`** to replay it; restore doesn't round-trip. immich (passes) uses dump-only path + `restore.post-hook` | **genuine RECIPE defect** | recipe PR: adopt immich-style postgres dump + `backupbot.restore.post-hook` replay | | mumble | `test_handshake…` RED | **ALL tiers GREEN** in isolation (×N) incl. handshake | **FLAKE (load/timing)** | handshake (TLS+ServerSync) doesn't complete within the 60s retry under heavy concurrent sweep load; fine isolated; canonical written green today | **load/concurrency FLAKE** | harness stabilization: stronger readiness gate before the custom tier / longer-or-smarter handshake retry | | bluesky-pds | warm promote `/xrpc/_health` → 000 | cold lifecycle GREEN; **warm promote 000 reproduces** | **genuine (deterministic)** — NOT a load/rate-limit flake | caddy on-demand TLS calls `http://app:3000/tls-check`; caddy resolves bare `app` to OTHER stacks' app endpoints on the shared `proxy` net (every stack aliases its main svc `app`), never bluesky's own internal app (10.0.3.3) → connection refused → no cert → 000 | **genuine ROUTING/RECIPE defect** (cross-stack `app`-alias collision) | recipe PR: give the PDS service a unique name/alias so caddy resolves only bluesky's app | | gitea | 3.5.3→3.6.0 warm advance doesn't promote | cold (incl fresh upgrade) GREEN; **warm advance crash-loops** | **genuine (deterministic)** | gitea 3.6.0/1.24.2 saves a JWT secret to `/etc/gitea/app.ini` on warm reattach; app.ini is a **read-only docker-config mount** → `read-only file system` FATA at LoadCommonSettings (pre-migration; 3.5.3 data intact). Cold passes (fresh secrets, no rewrite) | **genuine RECIPE defect** | recipe PR: render app.ini into the writable `config:/etc/gitea` volume (entrypoint) instead of a read-only docker config | | keycloak | de-enrolled (not tested) | code-verified (no run) | **genuine (structural)** | `canonical_domain("keycloak")` == `WARM_DOMAINS["keycloak"]` == `warm-keycloak.ci.commoninternet.net` EXACTLY → a data-warm canonical would collide with the live-warm OIDC provider | **HARNESS defect** (warm-domain namespace collision) | harness: collision-free `canonical_domain` for live-warm providers (`warm-canon-`), then enroll keycloak (WARM_CANONICAL=True) | ### HOW the Adversary cold-verifies each classification (run ONE recipe at a time, no concurrent load) Isolation invocation (per recipe `R` at latest tag `T`), from `/etc/cc-ci` on cc-ci: `git -C ~/.abra/recipes/R checkout -f --quiet T && env -u REF -u CCCI_QUICK -u MODE -u VERSION RECIPE=R CCCI_SKIP_FETCH=1 cc-ci-run runner/run_recipe_ci.py` Latest tags: discourse `0.8.1+3.5.0`, mattermost-lts `2.1.9+10.11.15`, mumble `1.0.0+v1.6.870-0`, bluesky-pds `0.3.0+v0.4.219`, gitea `3.6.0+1.24.2-rootless`. - **discourse** — EXPECT install/backup/restore/custom pass, upgrade fail on `test_head_runs_official_image_not_bitnamilegacy` + `test_sidekiq_service_dropped_by_head`. Confirm the overlay mismatch statically: `git -C ~/.abra/recipes/discourse show 0.8.1+3.5.0:compose.yml | grep -A1 ' app:'` and `... show main:compose.yml` both = `bitnamilegacy/discourse:3.5.0`; `grep -c 'sidekiq:'` = 1 in both. So the test's `discourse/discourse:3.5.3`/no-sidekiq expectation exists nowhere upstream. - **mattermost-lts** — EXPECT restore fail `relation "ci_marker" does not exist`. Confirm root cause statically: `git -C ~/.abra/recipes/mattermost-lts show 2.1.9+10.11.15:compose.yml | grep backupbot` shows pre-hook + `backup.path` but NO `restore.post-hook`; immich `git -C ~/.abra/recipes/immich show :compose.yml | grep backupbot` shows `restore.post-hook: /pg_backup.sh restore`. - **mumble** — EXPECT all tiers green (run 2–3× to confirm reproducibly green isolated). Canonical written green: `cat /var/lib/ci-warm/mumble/canonical.json`. - **bluesky-pds** — EXPECT cold green, WC5 promote `!! WC5 promote failed … warm-bluesky-pds … last status 0`. While the warm stack is up, confirm root cause: caddy logs `dial tcp 10.10.0.X:3000: connect: connection refused` for `app:3000/tls-check`; `docker exec getent hosts app` returns proxy IPs (10.10.0.X), the app's real internal IP is 10.0.3.x; `docker network inspect proxy | grep _app` shows many stacks aliasing `app`. (Tear down the orphaned warm-bluesky-pds stack + volumes after.) - **gitea** — REQUIRES idle canonical first: if warm-gitea is deployed, `docker stack rm warm-gitea_ci_commoninternet_net` (retains data+config volumes) so the advance reattaches from idle. EXPECT cold green, warm advance crash-loop with container log `LoadCommonSettings() [F] error saving JWT Secret … "/etc/gitea/app.ini": read-only file system`. Restore: leave warm-gitea undeployed (idle 3.5.3, volumes retained) — registry stays `3.5.3+1.24.2-rootless`. - **keycloak** — no run. Code-verify: `canonical.canonical_domain('keycloak')` → `warm.stable_domain('keycloak')` → `warm-keycloak.ci.commoninternet.net`; `warm.WARM_DOMAINS['keycloak']` == same string (runner/harness/canonical.py:42-44, warm.py:27-29,44-48). Live keycloak 200 on `/realms/master`. ### Node state left clean All isolation runs torn down; orphaned warm-bluesky-pds stack+volumes removed; warm-gitea restored to idle 3.5.3 (volumes retained, registry unchanged); only live warm-keycloak deployed (healthy). No `run_recipe_ci.py` processes. ## M1 — PASS @ 2026-06-18T01:18Z (REVIEW-redfix.md; all 6 classifications cold-verified CORRECT by Adversary's own isolation re-runs). No VETO. Cleared to M2. ## Phase: M2 — FIX + verify all six (IN PROGRESS) Fix designs locked in BACKLOG-redfix.md. Recipe PRs (mattermost-lts/bluesky/gitea) on git.autonomic.zone mirrors via the recipe mirror+PR flow, verified `!testme` (NEVER merge). Harness fixes (keycloak/mumble) on a cc-ci branch, verified via the harness. discourse: overlay-scope decision. Node now free for my deploys (Adversary done with M1). ### M2 fix tracker (updated 2026-06-18T05:53Z — ALL VERIFIED) | Recipe | Class | Fix | PR/branch + ref | Status | |---|---|---|---|---| | mattermost-lts | recipe defect | pg_backup.sh + `backupbot.restore.post-hook` (immich pattern) | mirror PR #1 `ci/pg-restore` @4ca7f418 | **VERIFIED** — !testme run #901 ALL tiers green incl `test_restore_returns_state` | | discourse | stale cc-ci overlay | recipe: bitnamilegacy->official discourse image migration + drop orphaned image-less sidekiq from compose.smtpauth.yml (F-redfix-1) | mirror PR #4 `discourse-official-image` @9ff5e19 | **VERIFIED** — own cold run `/tmp/redfix-discourse-m2verify.log` **level=5 of 5** (all tiers + lint R011 PASS); F-redfix-1 regression fixed | | keycloak | harness defect | collision-free `canonical_domain` (`warm-canon-` for WARM_DOMAINS recipes) + enroll | cc-ci branch `redfix-m2-harness` @61211db | **VERIFIED** — branch-checkout run promotes at warm-canon-keycloak; live warm-keycloak 200 throughout | | mumble | load/timing flake | harness: handshake readiness budget 60s->180s | cc-ci branch `redfix-m2-harness` @07fc6d4 | **VERIFIED** — branch-checkout run all tiers green incl handshake; budget active+non-regressing | | gitea | recipe defect | app.ini->staging `/etc/gitea/app.ini.init` + docker-setup seed-on-EMPTY + DOCKER_SETUP_SH_VERSION v3 | mirror PR #2 `ci/app-ini-writable` @a0f2db8 | **VERIFIED** (direct chaos-deploy; promote merge-gated — see below) | | bluesky-pds | recipe defect (routing) | caddy `{$APP_HOST}=${STACK_NAME}_app` (operator: NO rename) + CADDYFILE_VERSION v2 | mirror PR #4 `ci/warm-routing-alias` @4987ba9 | **VERIFIED** (direct chaos-deploy; promote merge-gated — see below) | cc-ci-side change verification: run from a checkout of `redfix-m2-harness` (CCCI_REPO=); never touches /etc/cc-ci main. `redfix-m2-harness` is now mumble+keycloak ONLY (bluesky needs no cc-ci change with the ${STACK_NAME}_app approach; the rename's exec-ref commit b96b8a4 was dropped). ## Gate: M2 — RE-CLAIMED, awaiting Adversary (2026-06-18T06:55Z; orig claim 05:53Z) **Re-claim delta (addresses Adversary M2 FAIL @06:42Z — finding F-redfix-1).** The first M2 verdict was FAIL on discourse ONLY (other 5 PASS, do-not-redo). F-redfix-1: the official-image migration dropped `sidekiq` from compose.yml but left a dangling image-less `sidekiq:` block in `compose.smtpauth.yml` → L5 lint R011 fail (run level=4) + broken SMTP-auth deploy. **FIXED** in PR #4 `discourse-official-image` @**9ff5e19** (force-pushed onto @53ba0910): dropped the orphaned `sidekiq:` block; the `app:` override already carries `DISCOURSE_SMTP_PASSWORD_FILE` + `smtp_password` secret (sidekiq is internal to the official image), so no SMTP coverage lost. `grep sidekiq compose*.yml` = 0. **VERIFIED two ways:** (1) the Adversary's exact lint.py repro flow at 9ff5e19 → **R011 ✅**; (2) my own full cold run `/tmp/redfix-discourse-m2verify.log` → `RUN SUMMARY ... level=5 of 5`, all tiers pass (install/upgrade/backup/restore/custom), `lint rung: pass`. Node clean: no discourse stack, NO discourse canonical (untagged migrated head correctly does not promote — should_promote tagged-gate), recipe reset to published tag 0.8.1+3.5.0. The other 5 fixes are unchanged since their Adversary PASS (keycloak, mumble, gitea, bluesky-pds, mattermost-lts) — no re-run needed. Adversary cold-verify for discourse: clone discourse @9ff5e19, run `RECIPE=discourse CCCI_SKIP_FETCH=1 … run_recipe_ci.py` → EXPECT level=5 of 5 (lint R011 ✅, all tiers pass, both upgrade-overlay tests `test_head_runs_official_image_not_bitnamilegacy` + `test_sidekiq_service_dropped_by_head` pass); OR the lint-only repro in F-redfix-1 → R011 ✅. `grep -c sidekiq ~/.abra/recipes/discourse/compose*.yml` @9ff5e19 = 0. --- ## Gate: M2 — original claim (2026-06-18T05:53Z) **WHAT (M2 DoD).** All six canon-sweep failures FIXED — each via a recipe PR or a harness improvement — and verified green. No recipe left as a standing exception. Nothing merged (operator merges). Per recipe: - **mattermost-lts** (recipe PR #1) — added `pg_backup.sh` + postgres `backupbot.restore.post-hook` so the logical dump round-trips on restore. - **discourse** (recipe PR #4) — migrated the head off deprecated `bitnamilegacy` to the official `discourse/discourse` image so the stale PR-faithfulness overlay (`test_head_runs_official_image…`, `test_sidekiq_service_dropped…`) passes on the migrated head (NOT a test-weakening). - **keycloak** (harness branch) — `canonical_domain` returns a collision-free `warm-canon-` for recipes in `warm.WARM_DOMAINS` (live-warm OIDC providers); keycloak enrolled (WARM_CANONICAL=True). - **mumble** (harness branch) — handshake readiness budget widened 60s->180s (load-flake stabilization). - **gitea** (recipe PR #2) — app.ini is now seeded into the WRITABLE `/etc/gitea` volume by docker-setup (`if [ ! -s /etc/gitea/app.ini ]`, seed-on-EMPTY) from the read-only staging config `app.ini.init`; `DOCKER_SETUP_SH_VERSION` v1->v3 forces the new docker-setup to re-mount. Gitea 1.24.2 can then persist its JWT secret (the M1 read-only-app.ini crash is gone). - **bluesky-pds** (recipe PR #4) — caddy resolves its OWN app via the fully-qualified swarm name `${STACK_NAME}_app` (caddy `{$APP_HOST}` env, set in the caddy service) instead of bare `app`, which collided with other stacks' `app` aliases on the shared `proxy` net. CADDYFILE_VERSION v1->v2. **HOW + EXPECTED + WHERE (Adversary cold-verify, one recipe at a time, no concurrent load):** - **mattermost-lts** — read-only artifact: `/var/lib/cc-ci-runs/901/` on cc-ci — all tiers pass, `junit/restore__cc-ci__test_restore.xml` testsuite failures=0, `test_restore_returns_state` pass. OR re-run !testme on PR #1 @4ca7f418. EXPECT restore green. - **discourse** — !testme on PR #4 @53ba0910 (run #849 green) OR run from a checkout of the migrated head: EXPECT install/backup/restore/custom + upgrade overlay all pass (head now official image). - **keycloak** — from a `redfix-m2-harness` @61211db checkout (CCCI_REPO=), run `RECIPE=keycloak CCCI_SKIP_FETCH=1 ... run_recipe_ci.py`. EXPECT all cold tiers pass + WC5 promote succeeds at domain `warm-canon-keycloak.ci.commoninternet.net` (NOT warm-keycloak); live `warm-keycloak.ci.commoninternet.net/realms/master` stays 200 throughout. Code: `canonical.py` canonical_domain returns warm-canon- for r in warm.WARM_DOMAINS. - **mumble** — from `redfix-m2-harness` @07fc6d4 checkout, run `RECIPE=mumble CCCI_SKIP_FETCH=1 …`. EXPECT all 5 tiers green incl `custom/test_protocol_handshake.py::test_handshake_completes_with_ channel_presence`; handshake budget = 36 attempts / 180s (was 60s). (Load-flake is not deterministically reproducible; this verifies the stabilization is applied, sound, non-weakening.) - **gitea** (recipe PR #2 @a0f2db8 on mirror branch `ci/app-ini-writable`) — DIRECT chaos-deploy proof (the harness WC5 promote is merge-gated, see NOTE). With the idle 3.5.3 canonical present: `cd ~/.abra/recipes/gitea && git checkout -f a0f2db8` then chaos-deploy onto the retained canonical volumes (0-byte app.ini = genuine pre-fix 3.5.3 state): `abra app deploy warm-gitea.ci.commoninternet.net -C -o -n`. EXPECT: service 1/1; the config volume's `app.ini` seeded 0->~1862 bytes (`INSTALL_LOCK = true`); `/api/v1/version` -> 200 {"version":"1.24.2"} and `/api/healthz` -> 200 (curl inside the app container); retained 3.5.3 data adopted (data dirs dated 2026-06-17T08:39); ZERO `read-only file system` crashes in `docker service logs` (M1 crashed here). Evidence: `/tmp/redfix-gitea-m2-directproof.log` on cc-ci. Teardown: `abra app undeploy … -n`, truncate the volume app.ini to 0 (restore pre-fix state). canonical.json stays 3.5.3 idle e6a1cc79. - **bluesky-pds** (recipe PR #4 @4987ba9 on mirror branch `ci/warm-routing-alias`) — DIRECT chaos-deploy proof (warm-promote is the only failing path; merge-gated). `git checkout -f 4987ba9`; generate secrets (`abra app secret generate warm-bluesky-pds.ci.commoninternet.net --all -m -C -o -n`) + insert a PLC rotation key (tests/bluesky-pds/install_steps.sh logic: 32-byte hex into pds_plc_rotation_key v1); **re-checkout 4987ba9 AFTER secret ops** (abra secret insert force-fetches+reverts the checkout); `abra app deploy warm-bluesky-pds.ci.commoninternet.net -C -o -n` (EXPECT `caddyfile: v1 -> v2`, NEW DEPLOYMENT 4987ba9). EXPECT: app+caddy 1/1; inside caddy `getent hosts warm-bluesky-pds_ci_commoninternet_net_app` -> a 10.0.x.x INTERNAL ip (own stack) while `getent hosts app` -> a 10.10.x.x proxy ip (foreign, the M1 collision); caddy log "certificate obtained successfully" with 0 "connection refused"; external `curl https://warm-bluesky-pds.ci. commoninternet.net/xrpc/_health` -> **200** {"version":"0.4.219"} (M1 was 000). Evidence: `/tmp/redfix-bluesky-m2-directproof.log`. Teardown: undeploy + remove volumes (caddy_data, pds_data) + secrets (no canonical, matching M1). **NOTE — gitea & bluesky end-to-end canonical-promote is OPERATOR-MERGE-GATED (not a shrug).** The harness WC5 promote does a recipe_checkout(published-tag)+non-chaos deploy, and BOTH run_recipe_ci.py:373 AND abra force-fetch `refs/tags/*` from upstream (abra.py:135 documents this), so any local move of the release tag to the fix commit is reverted to the PUBLISHED commit. The published 3.6.0 / 0.3.0 tags do NOT yet carry the fix (PR not merged — operator merges, per phase guardrail), so pre-merge the promote necessarily deploys the unfixed published release. Confirmed empirically: a full gitea harness run's WC5 promote deployed 357926f and crash-looped exactly like M1. The DIRECT chaos-deploy (chaos = deploy the working-tree checkout = the PR fix) is therefore the MAXIMAL + faithful pre-merge proof — it reproduces the EXACT M1 failing scenario (gitea: the retained canonical volumes; bluesky: warm-bluesky-pds on the shared proxy) and shows the fix resolves it. End-to-end canonical advance follows automatically once the operator merges PR #2 / #4 and the release tag carries the fix. This is NOT a standing exception — the defect is fixed + proven; only the registry-advance awaits the operator's merge (the phase's own "nothing merged" constraint). **WHERE (refs).** Recipe PRs on `git.autonomic.zone/recipe-maintainers/`: mattermost-lts `ci/pg-restore`@4ca7f418, discourse `discourse-official-image`@53ba0910, gitea `ci/app-ini-writable` @a0f2db8, bluesky-pds `ci/warm-routing-alias`@4987ba9. cc-ci harness branch `redfix-m2-harness`@07fc6d4 (keycloak 61211db + mumble 07fc6d4). Reasoning/dead-ends in JOURNAL-redfix.md. Node left clean (only infra + live warm-keycloak 200; gitea idle 3.5.3 volumes retained, canonical e6a1cc79 unchanged; no bluesky/test stacks/volumes/secrets; no run procs). ## Gate: M1 — PASS (above). **WHAT (M1 DoD).** All six canon-sweep failures investigated in ISOLATION (one recipe at a time, no concurrent sweep load), root-caused with first-hand evidence, and classified (flake vs genuine; recipe vs test vs warm-machinery vs load) — see the **M1 results table** + **HOW the Adversary cold-verifies** sections above. Summary: discourse = stale cc-ci overlay test (canon timeout/FATA root-cause was wrong); mattermost-lts = genuine recipe defect (no `backupbot.restore.post-hook`); mumble = load/timing FLAKE (2× isolation green); bluesky-pds = genuine routing defect (caddy↔app `app`-alias collision on shared proxy); gitea = genuine recipe defect (read-only app.ini config mount + 3.6.0 JWT save); keycloak = harness warm-domain namespace collision. NO "probably a flake" — every classification has an isolation re-run or code proof. **HOW + EXPECTED + WHERE.** Per-recipe cold-verify commands, expected outputs, and evidence paths are in the two sections above ("M1 results table" and "HOW the Adversary cold-verifies each classification"). Evidence logs on cc-ci: `/tmp/redfix-{discourse,mattermost-lts,mumble,mumble2,bluesky-pds,gitea2}.log`. Reasoning/dead-ends in JOURNAL-redfix.md. Node left clean (see "Node state left clean" above). ## Blocked (none)