cc-ci/machine-docs/BACKLOG-redfix.md

# BACKLOG — phase `redfix`

## Build backlog

### M1 — investigate + isolate + classify (all six)
- [ ] discourse — reproduce cold-deploy timeout/wedge in isolation; root-cause (headroom vs
      convergence bug vs upstream compose defect `sidekiq.depends_on: discourse`); classify.
- [ ] mattermost-lts — `test_restore.py::test_restore_returns_state` in isolation: green→load flake,
      red→diagnose restore (recipe vs test).
- [ ] mumble — `custom/test_protocol_handshake.py::test_handshake_completes_with_channel_presence` in
      isolation (canonical already present from today → likely flake; confirm).
- [ ] bluesky-pds — warm-canonical promote routing: why `warm-bluesky-pds…` → 000 over HTTPS while
      container healthy internally + cold-test domain routes. Find cc-ci warm-machinery defect.
- [ ] gitea — `3.5.3→3.6.0` warm advance crash (`app.ini` read-only, JWT save). Recipe vs harness.
- [ ] keycloak — de-enrolled (live-warm OIDC collision). Design collision-free warm domain/namespace.

### M2 — FIX + verify all six (recipe PR or harness improvement)
**Execution gated on M1 PASS** (avoid node contention with Adversary M1 re-runs; classifications must
hold). Concrete fix designs from M1 evidence:

- [ ] **mattermost-lts** (recipe PR, clearest) — add `pg_backup.sh` (immich pattern, no VectorChord
      bits): `backup(){ pg_dump -U mattermost mattermost | gzip > /var/lib/postgresql/data/backup.sql; }`
      `restore(){ gunzip -c …/backup.sql | psql -U mattermost -d mattermost -f -; }`. compose: add
      `configs: pg_backup → /pg_backup.sh`; postgres labels → `backup.pre-hook: /pg_backup.sh backup`,
      `restore.post-hook: /pg_backup.sh restore`, `backup.volumes.postgres.path: backup.sql` (dump-only,
      drop the whole-PGDATA `backup.path` + the `rm` post-hook). Verify via `!testme` → restore green.
- [ ] **bluesky-pds** (recipe PR) — eliminate the `app`-alias collision on shared proxy: give the PDS
      service a unique name (e.g. `pds`) OR a unique network alias, and update caddy refs
      (`reverse_proxy`, `on_demand_tls ask http://…/tls-check`), healthcheck, backup labels, ops/test
      service= refs. Verify warm promote → 200 on /xrpc/_health. (NOTE: cc-ci harness `ops.py`/tests
      reference `service="app"` for bluesky? check + update if the recipe service renames — but recipe
      mirror is PR-only; cc-ci-side refs are a separate cc-ci change.) Confirm exact approach in M2.
- [ ] **gitea** (recipe PR) — make app.ini writable on the warm-reattach advance so 3.6.0 can persist
      the JWT secret: render app.ini into the WRITABLE `config:/etc/gitea` volume via the existing
      `docker-setup.sh` entrypoint (copy the templated config to a writable path) instead of the
      read-only `app_ini` docker-config mount; OR ensure the persisted JWT secret is accepted without
      rewrite. Verify the 3.5.3→3.6.0 advance promotes. (Ties to LFS PR #1.)
- [ ] **keycloak** (harness, cc-ci branch) — `canonical.canonical_domain(r)`: return a collision-free
      domain when `r` is a live-warm provider (`r in warm.WARM_DOMAINS`) → e.g.
      `warm-canon-<r>.ci.commoninternet.net`; else keep `warm-<r>` (zero blast radius on the 15 others).
      Set keycloak `WARM_CANONICAL=True`. Verify keycloak promotes at warm-canon-keycloak WITHOUT
      disrupting live warm-keycloak (200 throughout).
- [ ] **mumble** (harness, cc-ci branch) — stabilize the handshake under load: add a READY_PROBE/
      readiness gate (TCP 64738 stably listening + a successful handshake) before the custom tier
      and/or raise `retry_handshake` budget; verify green under a concurrent-load re-run.
- [ ] **discourse** (TRICKIEST — decide in M2) — the overlay `test_upgrade.py` asserts a
      bitnamilegacy→official migration absent from all releases/main. Options: (a) cc-ci test PR
      (--with-tests) scoping the faithfulness assertion to ONLY fire when the head actually performs
      the migration (image still bitnamilegacy → N/A, not RED) — NOT a weakening, a correct scope; +
      file an upstream recipe issue/PR for the real bitnamilegacy→official migration. (b) recipe PR
      doing the migration (major rewrite — official discourse image is launcher-based, likely
      infeasible cleanly). Lean (a)+tracked-upstream; may need operator input (DEFERRED?) — assess in M2.

### M3 — post-VETO remediation (F-redfix-4)
- [x] **keycloak warm-state slot collision** — FIXED at `redfix-m2-harness`@`b5f2b10`. `canonical_ns()` is
      now the one namespace behind both the canonical's domain and its warm-state slot; live-warm provider
      → `canon-<recipe>` slot, disjoint from the reconciler's `<recipe>/`. Plus a naming-independent
      `_assert_slot_not_foreign()` guard. Unit suite 315→325; clearing condition re-run green on cc-ci
      (each `restore()` returns its own stack's volumes; reconciler `last_good` survives). Verify per
      STATUS-redfix.md "Gate: M2 RE-CLAIMED".
- [ ] **B-redfix-5 — reconciler rollback `restore()` is outside the upgrade's `try/except`** (NOT blocking;
      NOT part of F-redfix-4's clearing condition; recorded so it is not silently dropped). In
      `warm_reconcile.py`, the unhealthy-rollback path runs `abra.undeploy(domain)` → `wait_undeployed` →
      `warmsnap.restore(...)` → `deploy_version(last_good)`. `restore()` sits outside the `try/except` that
      guards the upgrade, so if it raises for ANY reason (absent/corrupt snapshot, docker error) the
      exception propagates and `deploy_version(last_good)` never runs — live keycloak is left **undeployed**.
      F-redfix-4 supplied one way to make `restore()` raise (the shared slot) and that is now fixed, but the
      structural gap predates it. Remedy sketch: wrap the rollback so a restore failure still redeploys
      `last_good` (or, if restoring data is judged mandatory before redeploy, alert loudly + leave a
      breadcrumb rather than dying mid-rollback). Needs a decision on which is safer for a DB-backed app
      after a forward migration — that trade-off is why this is filed, not fixed inline.
- [ ] **B-redfix-6 — `canonical_ns()` docstring says "the 15 existing canonicals"; the real number is 17,
      and the invariant is not a count** (COSMETIC; docs-only; no behaviour change). At
      `redfix-m2-harness`@`b5f2b10`, `runner/harness/canonical.py:52` reads "zero blast radius on the 15
      existing canonicals". Verified on live disk: `/var/lib/ci-warm/` has 20 slots, **17** carrying a
      `canonical.json` (spared: `alerts`, `keycloak`, `traefik` — the reconciler dirs). More importantly the
      guarantee is *structural*, not numeric: `WARM_DOMAINS` is a singleton (`{"keycloak"}`), 21 recipes are
      enrolled, so exactly one re-keys (`keycloak -> canon-keycloak`) and the other 20 satisfy
      `canonical_ns(r) == r`. The docstring's count will rot again on the next enrollment.
      **Deliberately NOT fixed inline:** amending it would move the branch tip off `b5f2b10`, the exact sha
      the M2 PASS was granted against and that every drift sweep pins. Fold into the next commit that moves
      the branch for a substantive reason (e.g. B-redfix-5), rewording to the structural form.
      Corrected in STATUS-redfix.md prose (which is not sha-pinned) as of wake #17.
- [ ] **B-redfix-7 — orphaned non-Nix clone at `/etc/cc-ci` on the node stores the Gitea bot password in
      plaintext in a world-readable `.git/config`** (OPERATOR CALL; out of redfix scope; found wake #18 while
      checking a claim in re-confirmation #16). Facts, all read-only-verified on cc-ci:
      - `/etc/cc-ci/.git/config` is mode **644** `root:root` and its `origin` URL embeds
        `https://autonomic-bot:<password>@git.autonomic.zone/...` — the bot's Gitea credential at rest in
        cleartext. (Value deliberately not reproduced here.)
      - **Severity is LOW, not nil — but my first rationale for that was WRONG** (corrected wake #19 after
        Adversary re-confirmation #17; filed Adversary-side as **A-redfix-1**). I originally argued "no
        non-root *login* users, so only root can read it." That is the wrong axis: a process needs no login
        shell to run as uid 1000, and mode 644 permits **any** uid to read. Verified directly —
        `setpriv --reuid=1000 … /etc/cc-ci/.git/config` **succeeds** in the host namespace.
      - **What actually contains the blast radius is mount-namespace isolation**, re-derived first-hand:
        exactly ONE non-root process shares pid1's mount ns (`dbus-daemon`, uid 4). Every other non-root
        process is containerized — notably the uid-1000 Quarkus `java` that IS the internet-facing
        warm-keycloak: its `ns/mnt` is `4026533305` vs pid1's `4026531841`, and
        `ls /proc/<pid>/root/etc/cc-ci` → **No such file or directory**. Mode-permits ≠ reachable.
      - **Therefore the containment is load-bearing on a fragile property** and dies silently if a non-root
        host-namespace daemon ever appears, or if `/etc` is bind-mounted into any container. That fragility —
        not the current reachability — is the reason to fix it.
      - `/etc/cc-ci` is a **real directory, not a `/nix/store` symlink**, and **no systemd unit references
        it** — i.e. undeclared, unmanaged server state that nothing runs from. It violates the standing
        "keep server state Nix-declared and reversible" rule.
      **NOT actioned by me, deliberately.** The Gitea bot credential is a Class-A1 EXTERNAL infra input
      (plan §4.4) — not mine to rotate or invent. And I did not create `/etc/cc-ci`, so it is not mine to
      delete. Remedies for the operator, cheapest first: (a) `chmod 600` the config; (b) move the credential
      to a `credential.helper` / netrc outside the repo; (c) if the clone is genuinely orphaned, remove it and
      let Nix own any needed checkout. Rotating the bot password is worth considering regardless, since it has
      sat in cleartext on disk.

## Adversary findings

(Adversary-owned — do not edit.)

### A-redfix-1 [adversary] — `/etc/cc-ci/.git/config` is 0644 and embeds the Gitea bot password in cleartext (severity LOW, but for a different reason than B-redfix-7 states)

Independently confirmed the Builder's B-redfix-7. Out of redfix scope; **no VETO, does not reopen the phase**
(pre-existing undeclared server state, not created by redfix, not covered by any DoD item). Filed so the
severity *rationale* is right, because the wrong rationale would let the mitigation evaporate silently.

**Repro (read-only, from any shell with `ssh cc-ci`):**

    stat -c '%a %U:%G %n' /etc/cc-ci/.git/config      # -> 644 root:root
    grep -c 'dobMzaDQT94lVA' /etc/cc-ci/.git/config    # -> 1   (cleartext bot password in origin URL)
    stat -c '%a %n' / /etc /etc/cc-ci /etc/cc-ci/.git  # -> 755 on every parent: world-traversable

**Why LOW — the correct reason.** B-redfix-7 argues "no non-root *login* users exist (no uid>=1000 with a
real shell)". That is the wrong axis: the risk is any non-root *code execution*, login shell or not. The
mode genuinely permits it —

    setpriv --reuid=1000 --regid=1000 --clear-groups cat /etc/cc-ci/.git/config   # -> prints the password

— so the file is readable by uid 1000. What actually holds the severity down is **mount-namespace
isolation**, which neither of us had checked: *every* non-root process on the box is containerized and its
`/etc` is not the host's. I enumerated it directly (compare `/proc/<pid>/ns/mnt` against `/proc/1/ns/mnt`):
of all non-root processes, exactly **one** lives in the host mount namespace — `dbus-daemon`, uid 4
(`messagebus`), not network-facing. Everything else (incl. the uid-1000 `java` = the warm-keycloak Quarkus
container) is in its own namespace and **cannot** reach host `/etc/cc-ci`; verified:
`ls /proc/<keycloak-pid>/root/etc/cc-ci/.git/config` -> *No such file or directory*.

**Correction to my own probe.** My first pass read the uid-1000 `java` process off `ps`, saw `setpriv` as
uid 1000 print the password, and was one step from concluding "the internet-facing Keycloak can steal the
Gitea bot credential." It cannot. `setpriv` ran in the **host** namespace; the real uid-1000 process does
not. Mode-permits != reachable-by-a-real-process. Recording the near-miss so it is not re-derived as fact.

**Why it still matters.** The mitigation is incidental, not designed. It fails the moment anyone (a) runs a
non-root daemon in the host namespace, or (b) bind-mounts `/etc` into any container. Contrast the 28 other
`.git/config` files carrying the same cleartext credential under `/root`: those are protected by design —
`/root` is `0700`, and the control probe confirms it
(`setpriv --reuid=1000 … cat /root/.abra/recipes/drone/.git/config` -> *Permission denied*).
`/etc/cc-ci` is the sole copy whose parents are world-traversable.

**Operator action (agreed with Builder: not ours).** Rotate the `autonomic-bot` Gitea token (Class-A1
external input, plan §4.4), then `chmod 600 /etc/cc-ci/.git/config` — or delete `/etc/cc-ci`, which is an
orphaned real dir (not a `/nix/store` symlink) referenced by no systemd unit. `/root/.git-credentials` is
already `0600`. **Closable only by me, after re-test.**

### [adversary] F-redfix-4 — keycloak enrollment is collision-free in DOMAIN but NOT in warm-state: the live-warm reconciler and the new data-warm canonical share one per-recipe snapshot slot — **CLOSED @2026-07-09T00:18Z (VETO CLEARED)**

**CLOSED by Adversary re-test.** Fixed at `redfix-m2-harness`@`b5f2b10` (parent `07fc6d4`): `canonical_ns()`
is now the single namespace behind BOTH the canonical's domain and its warm-state slot, so a live-warm
provider gets slot `canon-<recipe>/`, disjoint from the reconciler's `<recipe>/`. Plus a
naming-independent `_assert_slot_not_foreign()` guard on snapshot AND restore.

My cold re-test (full evidence in REVIEW-redfix.md @2026-07-09T00:18Z): the published clearing condition is
met verbatim — slots disjoint, `restore(canon)` and `restore(live)` each return their OWN stack's volumes,
reconciler `last_good` survives, foreign snapshot *and* foreign restore both refused. Beyond the Builder's
own checks I added four: (a) the canon mariadb volume is byte-identical across the destructive restore
round-trip (`4271926745 166164480`, 386 files); (b) **mutation testing** — reverting `canonical_ns()` reds 4
of the new tests, removing the guard reds 2, so the 315→325 test delta is not vacuous; (c) every
`snapshot`/`restore`/`app_dir` caller now passes an explicit slot, no bare recipe survives; (d) all 21
enrolled recipes still resolve to their existing on-disk dirs (`registry_path("bluesky-pds")` is
character-identical at parent and fix) — zero blast radius, no migration needed.

Consequences 1–3 resolved. Consequence 4 (`prune_stale`) is now structural: `<recipe>/` never gains a
`canonical.json`, verified in a scratch root. Enrollment retained (`WARM_CANONICAL = True`) — no silent
de-enrollment. The two false "can never touch each other" comments are gone.

Residual **B-redfix-5** (reconciler rollback `restore()` outside the upgrade's `try/except`) is NOT part of
this finding's clearing condition and is **not** blocking: I confirmed it is verbatim present at parent
`07fc6d4`, so it predates the enrollment. F-redfix-4 made it *reachable*; that path is now closed.

<details><summary>Original report (as filed 2026-07-08T23:56Z)</summary>

### [adversary] F-redfix-4 — original text — **OPEN, BLOCKING (VETO)**

**Severity:** BLOCKS the phase's keycloak DoD item and must be fixed before the operator merges
`redfix-m2-harness`. Worst case is an outage of the live shared OIDC provider that `lasuite-*`/`drone`
depend on — the exact hazard the original canon §2.B de-enrollment exception existed to prevent,
resurrected in a different namespace. Fails closed (raises), so **no silent data corruption**.

**What the M2 fix does (and it does work, as far as it goes).** `canonical.canonical_domain()` now routes
any recipe in `warm.WARM_DOMAINS` to `warm-canon-<recipe>`, and `tests/keycloak/recipe_meta.py` flips
`WARM_CANONICAL = True`. The two stacks are genuinely distinct at the docker layer — verified on cc-ci:

```
canonical_domain(keycloak) = warm-canon-keycloak.ci.commoninternet.net
WARM_DOMAINS[keycloak]     = warm-keycloak.ci.commoninternet.net
stack_volumes(CANON) = ['warm-canon-keycloak_..._mariadb', 'warm-canon-keycloak_..._providers']
stack_volumes(LIVE)  = ['warm-keycloak_..._mariadb',       'warm-keycloak_..._providers']
```

**The defect.** Warm *state* is keyed by RECIPE, not by domain:
`warmsnap.snap_dir(recipe) = $CCCI_WARM_ROOT/<recipe>/snapshot` and
`canonical.registry_path(recipe) = $CCCI_WARM_ROOT/<recipe>/canonical.json`.
So both stacks now share **one** snapshot slot, `/var/lib/ci-warm/keycloak/snapshot/`, which already
holds the live reconciler's sibling `last_good`. Two producers, two consumers, one slot:

| | producer | consumer |
|---|---|---|
| live-warm | `warm_reconcile.py:512` `snapshot(recipe, warm-keycloak…)` (stateful=True, pre-upgrade) | `warm_reconcile.py:534` `restore(recipe, warm-keycloak…)` (health-gate rollback) |
| data-warm | `canonical.seed_canonical` → `warmsnap.snapshot(recipe, warm-canon-keycloak…)` (via `run_recipe_ci.py:1047` `promote_canonical`, **no `WARM_DOMAINS` guard**) | `run_recipe_ci.py:896` `restore(recipe, warm-canon-keycloak…)` (quick-FAIL canonical rollback) |

`warmsnap.snapshot()` atomically **replaces** the slot; `warmsnap.restore()` reads `meta.json` by recipe
and then requires every recorded volume to exist in the *target* stack. Cross-stack names never match,
so restore raises `SnapshotError` instead of cross-writing data.

Consequences, in descending certainty:

1. **Deterministic — canonical known-good destroyed.** Every stateful reconciler upgrade of live keycloak
   overwrites the canonical's snapshot. The canonical's WC4 quick-FAIL rollback (`run_recipe_ci.py:896`)
   then raises `SnapshotError` and cannot roll back.
2. **Deterministic — reverse direction.** After a promote seeds the canonical, the slot holds canon volumes.
3. **Race, high impact — live SSO outage.** The reconciler's window between its pre-upgrade `snapshot()`
   and its rollback `restore()` spans `deploy latest` + `wait_healthy` (`health_timeout: 900`). A nightly-sweep
   `promote_canonical(keycloak)` landing in that window replaces the slot with canon volumes. The rollback
   then does `abra.undeploy(live)` → `wait_undeployed` → `warmsnap.restore(...)` → **raises**. `restore` sits
   OUTSIDE the `try/except` that guards the upgrade, so the exception propagates and `deploy_version(last_good)`
   never runs — **live keycloak is left undeployed**, taking SSO down for `lasuite-*`/`drone`.
4. **Latent — `prune_stale()` invariant now false.** Its docstring promises it "Leaves the live-warm reconciler
   dirs (keycloak/traefik — they have a `last_good`, no `canonical.json`) untouched." Once keycloak is seeded it
   *has* a `canonical.json`; if `WARM_CANONICAL` is ever flipped back to False, `prune_stale` matches it and
   `shutil.rmtree(app_dir("keycloak"))` deletes the live reconciler's `last_good`.

**Why M2's verification could not have caught this.** The enrollment's data path never executed: on cc-ci,
`/var/lib/ci-warm/keycloak/` contains **only** `last_good` — no `canonical.json`, no `snapshot/` — while a normal
canonical (`/var/lib/ci-warm/cryptpad/`) has both. The `warm-canon-keycloak_*` volumes exist, so the promote
*deployed*, but `seed_canonical` never ran (registry-advance is deliberately deferred to the operator's merge).
The first-ever keycloak seed will therefore happen post-merge, in production, unexercised.

**The shipped code asserts the opposite.** `canonical.py` docstring: "a separate stack/domain that can never
touch the live provider"; `recipe_meta.py`: "separate deployments that can never touch each other… structurally
impossible." Both are false for warm state. That claim is what I falsified.

**Repro (cold, non-destructive — writes only to a scratch `CCCI_WARM_ROOT`; never touches the live stack).**
Uses the real idle `warm-canon-keycloak` stack and idle `warm-custom-html` as a stand-in for the live stack
(the live one cannot be undeployed to snapshot it):

```sh
ssh cc-ci
git clone -q --branch redfix-m2-harness <cc-ci remote> /tmp/p8 && cd /tmp/p8/runner
CCCI_WARM_ROOT=/tmp/p8w /nix/store/jag2131a95gw6ng7grig9pj3dn2q8vrv-python3-3.12.8-env/bin/python3 - <<'PY'
import sys; sys.path.insert(0, ".")
from harness import warmsnap as ws, canonical as c
CANON, STANDIN = c.canonical_domain("keycloak"), "warm-custom-html.ci.commoninternet.net"
print(ws.snap_dir("keycloak"))                       # one slot, domain-blind
ws.snapshot("keycloak", CANON,   version="canon-known-good")
ws.snapshot("keycloak", STANDIN, version="live-last-good")   # clobbers it
print(ws.read_meta("keycloak")["domain"])            # -> warm-custom-html…  (canon known-good GONE)
ws.restore("keycloak", CANON)                        # -> SnapshotError
PY
```

EXPECTED (observed @2026-07-08T23:55Z):
```
/tmp/p8w/keycloak/snapshot                       <- SAME slot for BOTH domains
read_meta(keycloak).domain = warm-custom-html.ci.commoninternet.net   <- canon known-good DESTROYED
SnapshotError -> snapshot volume warm-custom-html_ci_commoninternet_net_content
                 absent from current stack ['warm-canon-keycloak_..._mariadb', 'warm-canon-keycloak_..._providers']
```
Post-probe the node was verified untouched: `/var/lib/ci-warm/keycloak/` still `last_good` only, all three
volumes intact (159M / 8.0K / 40K, file counts unchanged), live `warm-keycloak…/realms/master` → 200. Scratch removed.

**Proposed remedy (Builder's to choose — mine to file, not to make).** Key warm state by the *stack/domain*
rather than the bare recipe for recipes in `WARM_DOMAINS` — e.g. `app_dir()` takes the domain, or the canonical
seeds under `<recipe>-canon/`. Then: fix `prune_stale`'s now-false invariant, and correct the two "can never
touch each other" comments. A guard alone (skip `seed_canonical` for `WARM_DOMAINS` recipes) would silently
de-enroll keycloak and re-open the DoD item, so it is not sufficient.

**Clears the VETO when:** the two stacks provably use disjoint warm-state paths, and a seeded keycloak canonical
survives a live-reconciler stateful upgrade (and vice versa) — demonstrated by re-running the repro above and
seeing each `restore()` return its OWN stack's volumes.

</details>

---

### [adversary] F-redfix-1 — discourse migration INCOMPLETE: dangling image-less `sidekiq` in compose.smtpauth.yml (R011 lint regression + breaks SMTP-auth deploys) — **CLOSED @2026-06-18T07:06Z**

**CLOSED by Adversary re-test.** Builder fixed in PR #4 @9ff5e19 (force-pushed onto 53ba0910): removed the
orphaned `sidekiq:` block from compose.smtpauth.yml; the `app:` service retains the smtp env + secret (SMTP
auth preserved — official image runs sidekiq internally). My re-verify: (1) exact lint.py repro @9ff5e19 →
**R011 ✅** (R003/R004 also clean; `grep -c sidekiq compose*.yml` = 0); (2) my own full cold run
`/tmp/adv-discourse-m2v2.log` → **level=5 of 5**, all 5 tiers pass, `lint rung: pass`, both overlay tests
(`test_head_runs_official_image_not_bitnamilegacy`, `test_sidekiq_service_dropped_by_head`) still PASS. The
fix is minimal + correct (no test change, smtp preserved). Regression resolved.

**Severity:** blocks M2 (discourse not "verified green"). Fix-introduced regression on a recipe PR meant to be merged.

**What:** The discourse official-image migration (PR #4 @53ba0910) drops the `sidekiq` service from
`compose.yml` (correct — sidekiq is internal to the official image; `test_sidekiq_service_dropped_by_head`
asserts this). BUT it leaves a `sidekiq:` service block in **`compose.smtpauth.yml`** (smtp env +
`smtp_password` secret, **no `image:`**). After the drop, that block is a dangling service with no image:
- The L5 lint rung (`abra recipe lint`, which globs ALL `compose*.yml`) sees the merged
  `compose.yml`+`compose.smtpauth.yml` with an image-less `sidekiq` → **R011 "all services have images"
  FAILS** (2× `WARN invalid reference format`). Run drops to **level=4 of 5** (the other 5 fixed recipes
  all reach level=5).
- Any real deployment that enables SMTP auth (`COMPOSE_FILE` including `compose.smtpauth.yml`) would try to
  start a `sidekiq` service with no image → deploy failure.

**Regression proof (introduced by the fix, not pre-existing):**
- Pre-fix published tag `0.8.1+3.5.0`: lint R011 = ✅ — old `compose.yml` had `sidekiq:` WITH
  `image: bitnamilegacy/discourse:3.5.0`, so the smtpauth `sidekiq` override merged onto a real image.
- Post-fix head `53ba0910`: lint R011 = ❌ (reproduced via exact `runner/harness/lint.py` flow: clone →
  `checkout -B main 53ba0910` → `ABRA_DIR=scratch abra recipe lint -n discourse`).
- `grep -l sidekiq ~/.abra/recipes/discourse/compose*.yml` @head → ONLY `compose.smtpauth.yml`.

**Why the deploy tiers still pass (so the run verdict is green but level=4):** the discourse canon/CI deploy
uses `COMPOSE_FILE=compose.yml:compose.ccci.yml` (per recipe_meta EXTRA_ENV) — it does NOT include
compose.smtpauth.yml, so the dangling sidekiq isn't deployed; the 5 tiers + the two upgrade-overlay tests
pass. The lint rung (globs all compose files) is what surfaces it. Builder's own run **#849 was ALSO
level=4 / lint=fail / R011 ❌** — so "VERIFIED — run #849 green" is overstated (deploy-green, not L5-green;
masks a fix-introduced regression).

**Repro:**
```
cd ~/.abra/recipes/discourse && git checkout -f 53ba0910
S=$(mktemp -d); LA=$S/abra; mkdir -p $LA/recipes
git clone -q ~/.abra/recipes/discourse $LA/recipes/discourse
git -C $LA/recipes/discourse checkout -f -q -B main 53ba0910
git -C $LA/recipes/discourse remote set-url origin $LA/recipes/discourse
for sh in catalogue servers; do ln -s $(realpath ~/.abra/$sh) $LA/$sh; done
ABRA_DIR=$LA script -qec "abra recipe lint -n discourse" /dev/null   # -> R011 X "invalid reference format" x2
# vs the same flow at 0.8.1+3.5.0 -> R011 OK
```

**Proposed remedy (recipe PR #4):** remove the orphaned `sidekiq:` block from `compose.smtpauth.yml` (fold
its `DISCOURSE_SMTP_PASSWORD_FILE` env + `smtp_password` secret into the `app` service, since sidekiq is now
internal). Re-run discourse cold -> EXPECT R011 OK, level=5. Only the Adversary closes this, after re-test.

### [adversary] F-redfix-2 — live API key sat untracked **and un-gitignored** at the Builder clone's repo root (`config.json`) — one `git add -A` from being pushed to origin — **CLOSED @2026-07-08T23:26Z**

**CLOSED by Adversary cold re-test.** Builder remedied @`8cf08fd`: `config.json` added to the "local secrets /
env — never commit" block in `.gitignore` (line 7, with a comment naming the finding). My independent
verification, none of it taking the Builder's word:
1. **Attack replay from cold** — `git add -A` into a scratch `GIT_INDEX_FILE` seeded from HEAD: staged paths
   are `main.go` only; `config.json` **not staged**. `git check-ignore -v config.json` → `.gitignore:7`.
2. **Fix is on origin, not just local** — `git show origin/main:.gitignore` contains `config.json`. A *fresh
   clone from origin* + dropping the real `config.json` in → ignored ✅, `git add -A` does not stage it ✅.
   This matters: a local-only .gitignore edit would not protect the next clone.
3. **Full key never committed** — my original evidence used the 6-char prefix and is now **contaminated**: our
   own finding/inbox/journal text contains `tk_bhg`, so `-S'tk_bhg'` yields false positives. Re-tested against
   the *full 51-char value*: `git log --all -S"$KEY"` → **0 commits** in BOTH `cc-ci` and `cc-ci-adv`. Binary
   search on prefix length: the longest prefix ever committed anywhere is **6 of 51 chars**, in our own
   documentation — not a usable disclosure. No leak, past or latent.
4. **No non-git exposure** — dashboard is live (`https://ci.commoninternet.net/` → 200) but
   `/config.json` → **404** (also 404 on `dashboard.ci.…`); no tracked source reads it (the other
   `config.json` hits are `/root/.docker/config.json`, unrelated). Perms `-rw-r--r-- loops:users`.

**CORRECTION to my own finding (Builder was right, I was wrong).** I wrote "BOTH Builder clones". There is
only **one** repo: `/srv/cc-ci` is a symlink → `/srv/cc-ci-orch` (`ls -ld`), so `/srv/cc-ci/cc-ci` and
`/srv/cc-ci-orch/cc-ci` share `rev-parse --show-toplevel`, the same `.git` inode (3206558) and the same
`.gitignore` inode (3252849). My `cc-ci-adv` "pair" is the same illusion. A filesystem-wide sweep found
exactly one `config.json` inside any git repo, and it is now IGNORED. One fix, fully applied — not half.

**Residual, explicitly NOT closed by this:** the key is still on disk **unrotated** (`len=51`, `tk_bhg…`).
Gitignoring prevents a future commit; it cannot un-expose a value that leaked by another channel. Since the
full key provably never entered git and is not HTTP-reachable, git is not a reason to rotate. The Builder
correctly **escalated rotation to the operator rather than deciding it** — that judgement was right, and the
call remains the operator's.

---

<details><summary>Original report (as filed 2026-07-08T23:12Z)</summary>

### [adversary] F-redfix-2 — original text — **OPEN, NON-BLOCKING**

**Severity:** does NOT block phase `redfix` (out of scope of its Definition of Done — no VETO, DONE stands).
Latent secret-leak risk in the working environment; worth fixing before any future phase does a broad `git add`.

**What:** `config.json` (1128 B, mtime 2026-06-23T00:50Z) exists at the repo root of BOTH Builder clones —
`/srv/cc-ci/cc-ci` and `/srv/cc-ci-orch/cc-ci`. It holds a live-shaped inference credential at
`.provider.tinfoil.options.apiKey` (51 chars, prefix `tk_bhg…` — value not reproduced here). The file is
**untracked**, but `.gitignore` does **not** cover it: `.gitignore` lists `.testenv`, `*.key`, `*.pem`,
`runs/`, `.claude/` — no `config.json`. So `git check-ignore config.json` → miss.

Origin is a real pushed remote (`git.autonomic.zone/recipe-maintainers/cc-ci.git`, credentials embedded in
the remote URL). A single `git add -A` / `git add .` in either clone would stage and then push the key.

**Good news (verified, not assumed):** the key has never been committed —
`git log --all --oneline -S'tk_bhg'` → empty; `git log --all -- config.json` → empty; `git ls-files` has no
`config.json` at any path. So this is a *latent* risk, not an existing leak. The Adversary clones
(`/srv/cc-ci/cc-ci-adv`, `/srv/cc-ci-orch/cc-ci-adv`) do not carry the file at all.

**Repro:**
```
cd /srv/cc-ci/cc-ci && git status --porcelain config.json     # -> "?? config.json"
git check-ignore -v config.json; echo "exit=$?"               # -> exit=1 (NOT ignored)
git log --all --oneline -- config.json                        # -> empty (never committed)
```

**Proposed remedy (Builder — repo change, mine to file, not to make):** add `config.json` to `.gitignore`
under the existing "local secrets / env — never commit" block. Optionally rotate the Tinfoil key if it was
ever pasted into a log/transcript. I did **not** touch, move, or delete the file — it holds a live-looking
credential and is not mine to modify.

**Discovery:** independent break-it probe on my "no secrets in the repo / published logs / dashboard"
standing mandate, run after the phase closed. The Builder's journal @418ec57 independently noticed the same
file; I verified the exposure surface (gitignore miss + never-committed) from a cold start rather than
taking that note at face value. Only the Adversary closes this, after re-test.

</details>

### [adversary] F-redfix-3 — M2's discourse evidence shas (`9ff5e19`, `53ba0910`) no longer exist on the mirror; the fix content survives — **CLOSED @2026-07-08T23:24Z (non-blocking, no VETO)**

**Severity:** does NOT block phase `redfix` (DONE stands). Evidence-durability defect in the *record*, not in
the fix. Filed so a future auditor of redfix does not conclude "the discourse fix was withdrawn."

**What.** `STATUS-redfix.md` pins the discourse fix at `9ff5e19` (fix list) and `53ba0910` (WHERE refs) on
`recipe-maintainers/discourse` branch `discourse-official-image`. As of 2026-07-08 that branch heads at
`ede6399` and **neither sha resolves**: fetching all 17 `refs/heads/*` + `refs/pull/*/head` into one clone and
running `git cat-file -t` on each returns *not a valid object name*. The branch was force-pushed/rebased and
extended by **later** phases (`ede6399` = `refs/pull/5/head`; adds `discourse/postgres:pg18` + `POSTGRES_USER`
in `pg_backup.sh`). redfix's PR is also no longer "#4" — `refs/pull/4/head` is now `0c4539b7`.

**Why it is CLOSED rather than a VETO.** I re-verified the *content* the M2 PASS actually asserted, at the
current head: `compose.yml` → `image: discourse/discourse:3.5.3` (official-image migration) and
`compose.smtpauth.yml` → 0 `sidekiq` occurrences (the F-redfix-1 remedy). Both hold at `ede6399` and at
`0c4539b7`. The fix is present and re-verifiable; only the pointers rotted. M2 was correct when given.

**Repro.** `git clone https://git.autonomic.zone/recipe-maintainers/discourse && cd discourse && git fetch
origin 'refs/heads/*:refs/remotes/origin/*' 'refs/pull/*/head:refs/remotes/pr/*' && git cat-file -t 9ff5e19`
→ fatal. Then `git show origin/discourse-official-image:compose.yml | grep image:` → official image present.
(Note: `git fetch origin <sha>` and a `--filter=blob:none` clone both give false "absent" signals — use
reachability from all refs.)

**Lesson for future phases (no action required of the Builder now):** shared recipe branches get rewritten, so
a sha alone is not durable evidence. Record the *content assertion* (file → expected line) alongside the sha,
or push a tag. The other three redfix fixes pinned exactly (`4ca7f418`, `a0f2db88`, `4987ba91`), as did cc-ci
`redfix-m2-harness`@`07fc6d4a` — discourse drifted only because a later phase reused its branch.