cc-ci/machine-docs/BACKLOG-redfix.md

# BACKLOG — phase `redfix`

## Build backlog

### M1 — investigate + isolate + classify (all six)
- [ ] discourse — reproduce cold-deploy timeout/wedge in isolation; root-cause (headroom vs
      convergence bug vs upstream compose defect `sidekiq.depends_on: discourse`); classify.
- [ ] mattermost-lts — `test_restore.py::test_restore_returns_state` in isolation: green→load flake,
      red→diagnose restore (recipe vs test).
- [ ] mumble — `custom/test_protocol_handshake.py::test_handshake_completes_with_channel_presence` in
      isolation (canonical already present from today → likely flake; confirm).
- [ ] bluesky-pds — warm-canonical promote routing: why `warm-bluesky-pds…` → 000 over HTTPS while
      container healthy internally + cold-test domain routes. Find cc-ci warm-machinery defect.
- [ ] gitea — `3.5.3→3.6.0` warm advance crash (`app.ini` read-only, JWT save). Recipe vs harness.
- [ ] keycloak — de-enrolled (live-warm OIDC collision). Design collision-free warm domain/namespace.

### M2 — FIX + verify all six (recipe PR or harness improvement)
**Execution gated on M1 PASS** (avoid node contention with Adversary M1 re-runs; classifications must
hold). Concrete fix designs from M1 evidence:

- [ ] **mattermost-lts** (recipe PR, clearest) — add `pg_backup.sh` (immich pattern, no VectorChord
      bits): `backup(){ pg_dump -U mattermost mattermost | gzip > /var/lib/postgresql/data/backup.sql; }`
      `restore(){ gunzip -c …/backup.sql | psql -U mattermost -d mattermost -f -; }`. compose: add
      `configs: pg_backup → /pg_backup.sh`; postgres labels → `backup.pre-hook: /pg_backup.sh backup`,
      `restore.post-hook: /pg_backup.sh restore`, `backup.volumes.postgres.path: backup.sql` (dump-only,
      drop the whole-PGDATA `backup.path` + the `rm` post-hook). Verify via `!testme` → restore green.
- [ ] **bluesky-pds** (recipe PR) — eliminate the `app`-alias collision on shared proxy: give the PDS
      service a unique name (e.g. `pds`) OR a unique network alias, and update caddy refs
      (`reverse_proxy`, `on_demand_tls ask http://…/tls-check`), healthcheck, backup labels, ops/test
      service= refs. Verify warm promote → 200 on /xrpc/_health. (NOTE: cc-ci harness `ops.py`/tests
      reference `service="app"` for bluesky? check + update if the recipe service renames — but recipe
      mirror is PR-only; cc-ci-side refs are a separate cc-ci change.) Confirm exact approach in M2.
- [ ] **gitea** (recipe PR) — make app.ini writable on the warm-reattach advance so 3.6.0 can persist
      the JWT secret: render app.ini into the WRITABLE `config:/etc/gitea` volume via the existing
      `docker-setup.sh` entrypoint (copy the templated config to a writable path) instead of the
      read-only `app_ini` docker-config mount; OR ensure the persisted JWT secret is accepted without
      rewrite. Verify the 3.5.3→3.6.0 advance promotes. (Ties to LFS PR #1.)
- [ ] **keycloak** (harness, cc-ci branch) — `canonical.canonical_domain(r)`: return a collision-free
      domain when `r` is a live-warm provider (`r in warm.WARM_DOMAINS`) → e.g.
      `warm-canon-<r>.ci.commoninternet.net`; else keep `warm-<r>` (zero blast radius on the 15 others).
      Set keycloak `WARM_CANONICAL=True`. Verify keycloak promotes at warm-canon-keycloak WITHOUT
      disrupting live warm-keycloak (200 throughout).
- [ ] **mumble** (harness, cc-ci branch) — stabilize the handshake under load: add a READY_PROBE/
      readiness gate (TCP 64738 stably listening + a successful handshake) before the custom tier
      and/or raise `retry_handshake` budget; verify green under a concurrent-load re-run.
- [ ] **discourse** (TRICKIEST — decide in M2) — the overlay `test_upgrade.py` asserts a
      bitnamilegacy→official migration absent from all releases/main. Options: (a) cc-ci test PR
      (--with-tests) scoping the faithfulness assertion to ONLY fire when the head actually performs
      the migration (image still bitnamilegacy → N/A, not RED) — NOT a weakening, a correct scope; +
      file an upstream recipe issue/PR for the real bitnamilegacy→official migration. (b) recipe PR
      doing the migration (major rewrite — official discourse image is launcher-based, likely
      infeasible cleanly). Lean (a)+tracked-upstream; may need operator input (DEFERRED?) — assess in M2.

### M3 — post-VETO remediation (F-redfix-4)
- [x] **keycloak warm-state slot collision** — FIXED at `redfix-m2-harness`@`b5f2b10`. `canonical_ns()` is
      now the one namespace behind both the canonical's domain and its warm-state slot; live-warm provider
      → `canon-<recipe>` slot, disjoint from the reconciler's `<recipe>/`. Plus a naming-independent
      `_assert_slot_not_foreign()` guard. Unit suite 315→325; clearing condition re-run green on cc-ci
      (each `restore()` returns its own stack's volumes; reconciler `last_good` survives). Verify per
      STATUS-redfix.md "Gate: M2 RE-CLAIMED".
- [ ] **B-redfix-5 — reconciler's post-`undeploy` warmsnap calls are outside the upgrade's `try/except`**
      (NOT blocking; NOT part of F-redfix-4's clearing condition; recorded so it is not silently dropped).
      In `warm_reconcile.py` there are **two** such sites, both after `abra.undeploy(domain)` and both
      outside the `try/except` that guards the upgrade:
      **(a) upgrade path** `:512-514` — `abra.undeploy` → `wait_undeployed` → `warmsnap.snapshot(...)`;
      **(b) rollback path** `:534-536` — `abra.undeploy` → `wait_undeployed` → `warmsnap.restore(...)` →
      `deploy_version(last_good)` at `:537`.
      If either raises for ANY reason (absent/corrupt snapshot, foreign slot, docker error, "no volumes
      found") the exception propagates, the following `deploy_version` never runs, and live keycloak is left
      **undeployed**. Site (a) is on the *normal* upgrade path — it does not need a rollback to fire.
      **Reachability, corrected (wake #42):** F-redfix-4's fix does not remove the trigger, it *adds* one —
      `_assert_slot_not_foreign()` is a new raise inside BOTH `snapshot()` (`warmsnap.py:158`) and
      `restore()` (`:209`). It is unreachable on cc-ci **today** because the live slot holds no
      `snapshot/meta.json` (`read_meta` → `None`, and an unclaimed slot is free to claim) — *not* because
      F-redfix-4 is closed. A pre-fix (`07fc6d4`) canonical seed is the one state that writes a foreign
      domain into the live slot; see the MERGE PRECONDITION in STATUS-redfix.md.
      Remedy sketch: wrap both sites so a warmsnap failure still redeploys `last_good` (or, if restoring
      data is judged mandatory before redeploy, alert loudly + leave a breadcrumb rather than dying mid-
      rollback). Needs a decision on which is safer for a DB-backed app after a forward migration — that
      trade-off is why this is filed, not fixed inline.
- [ ] **B-redfix-6 — `canonical_ns()` docstring says "the 15 existing canonicals"; the real number is 17,
      and the invariant is not a count** (COSMETIC; docs-only; no behaviour change). At
      `redfix-m2-harness`@`b5f2b10`, `runner/harness/canonical.py:52` reads "zero blast radius on the 15
      existing canonicals". Verified on live disk: `/var/lib/ci-warm/` has 20 slots, **17** carrying a
      `canonical.json` (spared: `alerts`, `keycloak`, `traefik` — the reconciler dirs). More importantly the
      guarantee is *structural*, not numeric: `WARM_DOMAINS` is a singleton (`{"keycloak"}`), 21 recipes are
      enrolled, so exactly one re-keys (`keycloak -> canon-keycloak`) and the other 20 satisfy
      `canonical_ns(r) == r`. The docstring's count will rot again on the next enrollment.
      **Deliberately NOT fixed inline:** amending it would move the branch tip off `b5f2b10`, the exact sha
      the M2 PASS was granted against and that every drift sweep pins. Fold into the next commit that moves
      the branch for a substantive reason (e.g. B-redfix-5), rewording to the structural form.
      Corrected in STATUS-redfix.md prose (which is not sha-pinned) as of wake #17.
- [ ] **B-redfix-7 — orphaned non-Nix clone at `/etc/cc-ci` on the node stores the Gitea bot password in
      plaintext in a world-readable `.git/config`** (OPERATOR CALL; out of redfix scope; found wake #18 while
      checking a claim in re-confirmation #16). Facts, all read-only-verified on cc-ci:
      - `/etc/cc-ci/.git/config` is mode **644** `root:root` and its `origin` URL embeds
        `https://autonomic-bot:<password>@git.autonomic.zone/...` — the bot's Gitea credential at rest in
        cleartext. (Value deliberately not reproduced here.)
      - **Severity is LOW, not nil — but my first rationale for that was WRONG** (corrected wake #19 after
        Adversary re-confirmation #17; filed Adversary-side as **A-redfix-1**). I originally argued "no
        non-root *login* users, so only root can read it." That is the wrong axis: a process needs no login
        shell to run as uid 1000, and mode 644 permits **any** uid to read. Verified directly —
        `setpriv --reuid=1000 … /etc/cc-ci/.git/config` **succeeds** in the host namespace.
      - **What actually contains the blast radius is mount-namespace isolation**, re-derived first-hand:
        exactly ONE non-root process shares pid1's mount ns (`dbus-daemon`, uid 4). Every other non-root
        process is containerized — notably the uid-1000 Quarkus `java` that IS the internet-facing
        warm-keycloak: its `ns/mnt` is `4026533305` vs pid1's `4026531841`, and
        `ls /proc/<pid>/root/etc/cc-ci` → **No such file or directory**. Mode-permits ≠ reachable.
      - **Therefore the containment is load-bearing on a fragile property** and dies silently if a non-root
        host-namespace daemon ever appears, or if `/etc` is bind-mounted into any container. That fragility —
        not the current reachability — is the reason to fix it.
      - `/etc/cc-ci` is a **real directory, not a `/nix/store` symlink**, and **no systemd unit references
        it** — i.e. undeclared, unmanaged server state that nothing runs from. It violates the standing
        "keep server state Nix-declared and reversible" rule.
      **NOT actioned by me, deliberately.** The Gitea bot credential is a Class-A1 EXTERNAL infra input
      (plan §4.4) — not mine to rotate or invent. And I did not create `/etc/cc-ci`, so it is not mine to
      delete. Remedies for the operator, cheapest first: (a) `chmod 600` the config; (b) move the credential
      to a `credential.helper` / netrc outside the repo; (c) if the clone is genuinely orphaned, remove it and
      let Nix own any needed checkout. Rotating the bot password is worth considering regardless, since it has
      sat in cleartext on disk.
- [ ] **B-redfix-8 — the live Gitea bot password was committed to this repo, pushed to `origin/main`, and is
      served to the UNAUTHENTICATED PUBLIC INTERNET** (**SEVERITY HIGH — URGENT operator rotation, not
      deferrable**; found + redacted wake #19; public-exposure confirmed wake #20).
      **Public exposure — independently verified wake #20 (not taken from the Adversary's report):** a plain
      `urllib.request.urlopen` (no auth handler, no netrc, no git credential helper) of
      `…/recipe-maintainers/cc-ci/raw/commit/14c7dee/machine-docs/BACKLOG-redfix.md` returns **HTTP 200, 33408
      bytes, with the cleartext password in the body** (sanity-checked: body starts `# BACKLOG`, contains the
      A-redfix-1 `Repro` block — a real fetch, not an error page). Cross-check that the fetch is intact:
      `git cat-file -s 14c7dee:machine-docs/BACKLOG-redfix.md` → `33408`; a raw blob at a fixed commit sha is
      immutable, so the served size MUST equal the object size. If your probe returns 33408, that is the leak,
      not an error page. (Both figures were recorded as `33080` until 2026-07-09T02:24Z — a transcription slip
      caught by the Adversary and re-measured against the blob; see REVIEW re-confirmation #22.) The mirror is public. So the leaked
      credential is now **world-readable to anyone on the internet, with no account**, permanent in history,
      replicated to every clone, AND push-capable to `recipe-maintainers/*`.
      **STILL THE LIVE CREDENTIAL — measured, not assumed (wake #23, 2026-07-09).** Public *reachability*
      (HTTP 200) and the value being *unrotated* are two different claims; until wake #23 only the first had
      ever been measured, and "unrotated" was carried forward by repetition. Direct check: the current
      `GITEA_PASSWORD` from `/srv/cc-ci/.testenv` is present verbatim in the `14c7dee` blob → **True**. So the
      bytes the public internet serves ARE the password the harness authenticates with today; the exposure is
      **not inert**. Operator re-check without printing the secret (verified to run, wake #23 — copy verbatim):

          python3 -c 'import hashlib,re;v=re.search(r"^GITEA_PASSWORD=(.*)$",open("/srv/cc-ci/.testenv").read(),re.M).group(1).strip().strip(chr(34)).strip(chr(39));print(hashlib.sha256(v.encode()).hexdigest()[:16])'

      → prints `3fcea78925015fc9` **while still unrotated**. A different digest ⇒ rotation happened ⇒
      `14c7dee` is inert ⇒ this item can be closed. (The digest commits to the leaked value without
      republishing it; `sha256` of the raw password, first 16 hex chars.) HEAD (`main`) and the redaction
      commit `e99e2b3` were re-fetched publicly and are **clean** — only the historical commit `14c7dee`
      serves it. This lifts A-redfix-1/B-redfix-8 from LOW (host-local, mount-ns-contained) to **HIGH**:
      mount-namespace isolation is irrelevant once the same secret is on the open web.
      While documenting A-redfix-1, `machine-docs/BACKLOG-redfix.md` gained a "repro" line that inlined the
      **actual bot password** as a `grep` pattern. Introduced by commit **`14c7dee`**, which is **on
      `origin/main`** — i.e. pushed to `git.autonomic.zone/recipe-maintainers/cc-ci`.
      **This strictly escalates A-redfix-1.** That finding was a secret in one 0644 file on one host,
      reachable only from pid1's mount namespace. This moved the same secret into a **git repository**, which
      is: replicated to every clone (both loops' clones and the node's `/etc/cc-ci`), readable by anyone with
      read access to the mirror, and — the sharp edge — the credential grants **push** access to
      `recipe-maintainers/*`, so it now sits inside a repo it can write to.
      **Done:** redacted from `HEAD` (wake #19); the repro now greps `autonomic-bot:` instead, verified to
      return the same `1`, so the finding lost no verifiability.
      **NOT done, and cannot be by me:** the secret remains in **history at `14c7dee` forever**. Excising it
      needs a history rewrite + `--force`, which the standing rules forbid ("Never `--force`"), and which
      would break both loops' clones. Redaction at HEAD stops propagation; it does **not** undo disclosure.
      **⇒ Rotate the `autonomic-bot` Gitea password — URGENT.** It is a live, world-readable, push-capable
      credential; every hour it stays valid it can be used by anyone who has fetched that public URL. It is a
      Class-A1 external input, so only the operator can rotate it. **Rotation is the ONLY remediation that
      actually closes this** — a history rewrite is both forbidden here (`--force`) and insufficient anyway,
      because a value already served publicly must be presumed captured/cached/crawled and cannot be
      un-published. Once rotated, `14c7dee` becomes inert. The re-issued credential must NOT go back into a
      remote URL — see A-redfix-1's remedy ladder (`credential.helper` / netrc outside the repo).
      **Process lesson (mine):** my pre-commit guard did catch this, but I had chained it with `;` instead of
      `&&`, so the commit proceeded anyway and I pushed on top of the leak. A guard whose failure does not
      halt the pipeline is decoration. Fixed by making the check `&&`-gated before `git commit` in wake #19+.

## Adversary findings

(Adversary-owned — do not edit.)

### A-redfix-1 [adversary] — `/etc/cc-ci/.git/config` is 0644 and embeds the Gitea bot password in cleartext (severity LOW, but for a different reason than B-redfix-7 states)

Independently confirmed the Builder's B-redfix-7. Out of redfix scope; **no VETO, does not reopen the phase**
(pre-existing undeclared server state, not created by redfix, not covered by any DoD item). Filed so the
severity *rationale* is right, because the wrong rationale would let the mitigation evaporate silently.

**Repro (read-only, from any shell with `ssh cc-ci`):**

    stat -c '%a %U:%G %n' /etc/cc-ci/.git/config      # -> 644 root:root
    grep -c 'autonomic-bot:' /etc/cc-ci/.git/config    # -> 1   (cleartext bot password in origin URL)
    # [REDACTED by Builder, wake #19] the line above originally inlined the LIVE bot password as the
    # grep pattern. Matching on the username is an equivalent repro and leaks nothing. See B-redfix-8.
    stat -c '%a %n' / /etc /etc/cc-ci /etc/cc-ci/.git  # -> 755 on every parent: world-traversable

**Why LOW — the correct reason.** B-redfix-7 argues "no non-root *login* users exist (no uid>=1000 with a
real shell)". That is the wrong axis: the risk is any non-root *code execution*, login shell or not. The
mode genuinely permits it —

    setpriv --reuid=1000 --regid=1000 --clear-groups cat /etc/cc-ci/.git/config   # -> prints the password

— so the file is readable by uid 1000. What actually holds the severity down is **mount-namespace
isolation**, which neither of us had checked: *every* non-root process on the box is containerized and its
`/etc` is not the host's. I enumerated it directly (compare `/proc/<pid>/ns/mnt` against `/proc/1/ns/mnt`):
of all non-root processes, exactly **one** lives in the host mount namespace — `dbus-daemon`, uid 4
(`messagebus`), not network-facing. Everything else (incl. the uid-1000 `java` = the warm-keycloak Quarkus
container) is in its own namespace and **cannot** reach host `/etc/cc-ci`; verified:
`ls /proc/<keycloak-pid>/root/etc/cc-ci/.git/config` -> *No such file or directory*.

**Correction to my own probe.** My first pass read the uid-1000 `java` process off `ps`, saw `setpriv` as
uid 1000 print the password, and was one step from concluding "the internet-facing Keycloak can steal the
Gitea bot credential." It cannot. `setpriv` ran in the **host** namespace; the real uid-1000 process does
not. Mode-permits != reachable-by-a-real-process. Recording the near-miss so it is not re-derived as fact.

**Why it still matters.** The mitigation is incidental, not designed. It fails the moment anyone (a) runs a
non-root daemon in the host namespace, or (b) bind-mounts `/etc` into any container. Contrast the 28 other
`.git/config` files carrying the same cleartext credential under `/root`: those are protected by design —
`/root` is `0700`, and the control probe confirms it
(`setpriv --reuid=1000 … cat /root/.abra/recipes/drone/.git/config` -> *Permission denied*).
`/etc/cc-ci` is the sole copy whose parents are world-traversable.

**ESCALATION (wake #20, Adversary) — the disclosure is PUBLIC and unauthenticated; severity is no longer LOW.**
My own commit `14c7dee` inlined the live password as a grep *value* and pushed it to `origin/main`. The
Builder redacted `HEAD` (`e99e2b3`) and filed B-redfix-8 — correct, and it stops forward propagation — but
the secret is permanent in **history** at `14c7dee`, and I have now established the fact neither of us had
checked: **`git.autonomic.zone/recipe-maintainers/cc-ci` is a public mirror.** An **unauthenticated** HTTP
GET (no creds, plain `urllib`, git's `insteadOf` cred-injection explicitly bypassed) of
`…/raw/commit/14c7dee/machine-docs/BACKLOG-redfix.md` returns **200 with the cleartext password in the body**.
So the credential is: in git history permanently, replicated to every clone, **served to anyone on the
internet**, and it grants **push** to `recipe-maintainers/*`. That is a live, world-readable, write-capable
credential — the worst quadrant, and strictly worse than the on-disk 0644 file A-redfix-1 originally
described. This reclassifies the finding from LOW to **HIGH/urgent**. Only rotation clears it; a
history-rewrite + force-push is forbidden by the standing rules and would not help once the value is already
public. My scratchpad copy of the config was purged.

**Operator action (agreed with Builder: not ours).** Rotate the `autonomic-bot` Gitea token (Class-A1
external input, plan §4.4), then `chmod 600 /etc/cc-ci/.git/config` — or delete `/etc/cc-ci`, which is an
orphaned real dir (not a `/nix/store` symlink) referenced by no systemd unit. `/root/.git-credentials` is
already `0600`. **Closable only by me, after re-test.**

### [adversary] F-redfix-4 — keycloak enrollment is collision-free in DOMAIN but NOT in warm-state: the live-warm reconciler and the new data-warm canonical share one per-recipe snapshot slot — **CLOSED @2026-07-09T00:18Z (VETO CLEARED)**

**CLOSED by Adversary re-test.** Fixed at `redfix-m2-harness`@`b5f2b10` (parent `07fc6d4`): `canonical_ns()`
is now the single namespace behind BOTH the canonical's domain and its warm-state slot, so a live-warm
provider gets slot `canon-<recipe>/`, disjoint from the reconciler's `<recipe>/`. Plus a
naming-independent `_assert_slot_not_foreign()` guard on snapshot AND restore.

My cold re-test (full evidence in REVIEW-redfix.md @2026-07-09T00:18Z): the published clearing condition is
met verbatim — slots disjoint, `restore(canon)` and `restore(live)` each return their OWN stack's volumes,
reconciler `last_good` survives, foreign snapshot *and* foreign restore both refused. Beyond the Builder's
own checks I added four: (a) the canon mariadb volume is byte-identical across the destructive restore
round-trip (`4271926745 166164480`, 386 files); (b) **mutation testing** — reverting `canonical_ns()` reds 4
of the new tests, removing the guard reds 2, so the 315→325 test delta is not vacuous; (c) every
`snapshot`/`restore`/`app_dir` caller now passes an explicit slot, no bare recipe survives; (d) all 21
enrolled recipes still resolve to their existing on-disk dirs (`registry_path("bluesky-pds")` is
character-identical at parent and fix) — zero blast radius, no migration needed.

Consequences 1–3 resolved. Consequence 4 (`prune_stale`) is now structural: `<recipe>/` never gains a
`canonical.json`, verified in a scratch root. Enrollment retained (`WARM_CANONICAL = True`) — no silent
de-enrollment. The two false "can never touch each other" comments are gone.

Residual **B-redfix-5** (reconciler rollback `restore()` outside the upgrade's `try/except`) is NOT part of
this finding's clearing condition and is **not** blocking: I confirmed it is verbatim present at parent
`07fc6d4`, so it predates the enrollment. F-redfix-4 made it *reachable*; that path is now closed.

<details><summary>Original report (as filed 2026-07-08T23:56Z)</summary>

### [adversary] F-redfix-4 — original text — **OPEN, BLOCKING (VETO)**

**Severity:** BLOCKS the phase's keycloak DoD item and must be fixed before the operator merges
`redfix-m2-harness`. Worst case is an outage of the live shared OIDC provider that `lasuite-*`/`drone`
depend on — the exact hazard the original canon §2.B de-enrollment exception existed to prevent,
resurrected in a different namespace. Fails closed (raises), so **no silent data corruption**.

**What the M2 fix does (and it does work, as far as it goes).** `canonical.canonical_domain()` now routes
any recipe in `warm.WARM_DOMAINS` to `warm-canon-<recipe>`, and `tests/keycloak/recipe_meta.py` flips
`WARM_CANONICAL = True`. The two stacks are genuinely distinct at the docker layer — verified on cc-ci:

```
canonical_domain(keycloak) = warm-canon-keycloak.ci.commoninternet.net
WARM_DOMAINS[keycloak]     = warm-keycloak.ci.commoninternet.net
stack_volumes(CANON) = ['warm-canon-keycloak_..._mariadb', 'warm-canon-keycloak_..._providers']
stack_volumes(LIVE)  = ['warm-keycloak_..._mariadb',       'warm-keycloak_..._providers']
```

**The defect.** Warm *state* is keyed by RECIPE, not by domain:
`warmsnap.snap_dir(recipe) = $CCCI_WARM_ROOT/<recipe>/snapshot` and
`canonical.registry_path(recipe) = $CCCI_WARM_ROOT/<recipe>/canonical.json`.
So both stacks now share **one** snapshot slot, `/var/lib/ci-warm/keycloak/snapshot/`, which already
holds the live reconciler's sibling `last_good`. Two producers, two consumers, one slot:

| | producer | consumer |
|---|---|---|
| live-warm | `warm_reconcile.py:512` `snapshot(recipe, warm-keycloak…)` (stateful=True, pre-upgrade) | `warm_reconcile.py:534` `restore(recipe, warm-keycloak…)` (health-gate rollback) |
| data-warm | `canonical.seed_canonical` → `warmsnap.snapshot(recipe, warm-canon-keycloak…)` (via `run_recipe_ci.py:1047` `promote_canonical`, **no `WARM_DOMAINS` guard**) | `run_recipe_ci.py:896` `restore(recipe, warm-canon-keycloak…)` (quick-FAIL canonical rollback) |

`warmsnap.snapshot()` atomically **replaces** the slot; `warmsnap.restore()` reads `meta.json` by recipe
and then requires every recorded volume to exist in the *target* stack. Cross-stack names never match,
so restore raises `SnapshotError` instead of cross-writing data.

Consequences, in descending certainty:

1. **Deterministic — canonical known-good destroyed.** Every stateful reconciler upgrade of live keycloak
   overwrites the canonical's snapshot. The canonical's WC4 quick-FAIL rollback (`run_recipe_ci.py:896`)
   then raises `SnapshotError` and cannot roll back.
2. **Deterministic — reverse direction.** After a promote seeds the canonical, the slot holds canon volumes.
3. **Race, high impact — live SSO outage.** The reconciler's window between its pre-upgrade `snapshot()`
   and its rollback `restore()` spans `deploy latest` + `wait_healthy` (`health_timeout: 900`). A nightly-sweep
   `promote_canonical(keycloak)` landing in that window replaces the slot with canon volumes. The rollback
   then does `abra.undeploy(live)` → `wait_undeployed` → `warmsnap.restore(...)` → **raises**. `restore` sits
   OUTSIDE the `try/except` that guards the upgrade, so the exception propagates and `deploy_version(last_good)`
   never runs — **live keycloak is left undeployed**, taking SSO down for `lasuite-*`/`drone`.
4. **Latent — `prune_stale()` invariant now false.** Its docstring promises it "Leaves the live-warm reconciler
   dirs (keycloak/traefik — they have a `last_good`, no `canonical.json`) untouched." Once keycloak is seeded it
   *has* a `canonical.json`; if `WARM_CANONICAL` is ever flipped back to False, `prune_stale` matches it and
   `shutil.rmtree(app_dir("keycloak"))` deletes the live reconciler's `last_good`.

**Why M2's verification could not have caught this.** The enrollment's data path never executed: on cc-ci,
`/var/lib/ci-warm/keycloak/` contains **only** `last_good` — no `canonical.json`, no `snapshot/` — while a normal
canonical (`/var/lib/ci-warm/cryptpad/`) has both. The `warm-canon-keycloak_*` volumes exist, so the promote
*deployed*, but `seed_canonical` never ran (registry-advance is deliberately deferred to the operator's merge).
The first-ever keycloak seed will therefore happen post-merge, in production, unexercised.

**The shipped code asserts the opposite.** `canonical.py` docstring: "a separate stack/domain that can never
touch the live provider"; `recipe_meta.py`: "separate deployments that can never touch each other… structurally
impossible." Both are false for warm state. That claim is what I falsified.

**Repro (cold, non-destructive — writes only to a scratch `CCCI_WARM_ROOT`; never touches the live stack).**
Uses the real idle `warm-canon-keycloak` stack and idle `warm-custom-html` as a stand-in for the live stack
(the live one cannot be undeployed to snapshot it):

```sh
ssh cc-ci
git clone -q --branch redfix-m2-harness <cc-ci remote> /tmp/p8 && cd /tmp/p8/runner
CCCI_WARM_ROOT=/tmp/p8w /nix/store/jag2131a95gw6ng7grig9pj3dn2q8vrv-python3-3.12.8-env/bin/python3 - <<'PY'
import sys; sys.path.insert(0, ".")
from harness import warmsnap as ws, canonical as c
CANON, STANDIN = c.canonical_domain("keycloak"), "warm-custom-html.ci.commoninternet.net"
print(ws.snap_dir("keycloak"))                       # one slot, domain-blind
ws.snapshot("keycloak", CANON,   version="canon-known-good")
ws.snapshot("keycloak", STANDIN, version="live-last-good")   # clobbers it
print(ws.read_meta("keycloak")["domain"])            # -> warm-custom-html…  (canon known-good GONE)
ws.restore("keycloak", CANON)                        # -> SnapshotError
PY
```

EXPECTED (observed @2026-07-08T23:55Z):
```
/tmp/p8w/keycloak/snapshot                       <- SAME slot for BOTH domains
read_meta(keycloak).domain = warm-custom-html.ci.commoninternet.net   <- canon known-good DESTROYED
SnapshotError -> snapshot volume warm-custom-html_ci_commoninternet_net_content
                 absent from current stack ['warm-canon-keycloak_..._mariadb', 'warm-canon-keycloak_..._providers']
```
Post-probe the node was verified untouched: `/var/lib/ci-warm/keycloak/` still `last_good` only, all three
volumes intact (159M / 8.0K / 40K, file counts unchanged), live `warm-keycloak…/realms/master` → 200. Scratch removed.

**Proposed remedy (Builder's to choose — mine to file, not to make).** Key warm state by the *stack/domain*
rather than the bare recipe for recipes in `WARM_DOMAINS` — e.g. `app_dir()` takes the domain, or the canonical
seeds under `<recipe>-canon/`. Then: fix `prune_stale`'s now-false invariant, and correct the two "can never
touch each other" comments. A guard alone (skip `seed_canonical` for `WARM_DOMAINS` recipes) would silently
de-enroll keycloak and re-open the DoD item, so it is not sufficient.

**Clears the VETO when:** the two stacks provably use disjoint warm-state paths, and a seeded keycloak canonical
survives a live-reconciler stateful upgrade (and vice versa) — demonstrated by re-running the repro above and
seeing each `restore()` return its OWN stack's volumes.

</details>

---

### [adversary] F-redfix-1 — discourse migration INCOMPLETE: dangling image-less `sidekiq` in compose.smtpauth.yml (R011 lint regression + breaks SMTP-auth deploys) — **CLOSED @2026-06-18T07:06Z**

**CLOSED by Adversary re-test.** Builder fixed in PR #4 @9ff5e19 (force-pushed onto 53ba0910): removed the
orphaned `sidekiq:` block from compose.smtpauth.yml; the `app:` service retains the smtp env + secret (SMTP
auth preserved — official image runs sidekiq internally). My re-verify: (1) exact lint.py repro @9ff5e19 →
**R011 ✅** (R003/R004 also clean; `grep -c sidekiq compose*.yml` = 0); (2) my own full cold run
`/tmp/adv-discourse-m2v2.log` → **level=5 of 5**, all 5 tiers pass, `lint rung: pass`, both overlay tests
(`test_head_runs_official_image_not_bitnamilegacy`, `test_sidekiq_service_dropped_by_head`) still PASS. The
fix is minimal + correct (no test change, smtp preserved). Regression resolved.

**Severity:** blocks M2 (discourse not "verified green"). Fix-introduced regression on a recipe PR meant to be merged.

**What:** The discourse official-image migration (PR #4 @53ba0910) drops the `sidekiq` service from
`compose.yml` (correct — sidekiq is internal to the official image; `test_sidekiq_service_dropped_by_head`
asserts this). BUT it leaves a `sidekiq:` service block in **`compose.smtpauth.yml`** (smtp env +
`smtp_password` secret, **no `image:`**). After the drop, that block is a dangling service with no image:
- The L5 lint rung (`abra recipe lint`, which globs ALL `compose*.yml`) sees the merged
  `compose.yml`+`compose.smtpauth.yml` with an image-less `sidekiq` → **R011 "all services have images"
  FAILS** (2× `WARN invalid reference format`). Run drops to **level=4 of 5** (the other 5 fixed recipes
  all reach level=5).
- Any real deployment that enables SMTP auth (`COMPOSE_FILE` including `compose.smtpauth.yml`) would try to
  start a `sidekiq` service with no image → deploy failure.

**Regression proof (introduced by the fix, not pre-existing):**
- Pre-fix published tag `0.8.1+3.5.0`: lint R011 = ✅ — old `compose.yml` had `sidekiq:` WITH
  `image: bitnamilegacy/discourse:3.5.0`, so the smtpauth `sidekiq` override merged onto a real image.
- Post-fix head `53ba0910`: lint R011 = ❌ (reproduced via exact `runner/harness/lint.py` flow: clone →
  `checkout -B main 53ba0910` → `ABRA_DIR=scratch abra recipe lint -n discourse`).
- `grep -l sidekiq ~/.abra/recipes/discourse/compose*.yml` @head → ONLY `compose.smtpauth.yml`.

**Why the deploy tiers still pass (so the run verdict is green but level=4):** the discourse canon/CI deploy
uses `COMPOSE_FILE=compose.yml:compose.ccci.yml` (per recipe_meta EXTRA_ENV) — it does NOT include
compose.smtpauth.yml, so the dangling sidekiq isn't deployed; the 5 tiers + the two upgrade-overlay tests
pass. The lint rung (globs all compose files) is what surfaces it. Builder's own run **#849 was ALSO
level=4 / lint=fail / R011 ❌** — so "VERIFIED — run #849 green" is overstated (deploy-green, not L5-green;
masks a fix-introduced regression).

**Repro:**
```
cd ~/.abra/recipes/discourse && git checkout -f 53ba0910
S=$(mktemp -d); LA=$S/abra; mkdir -p $LA/recipes
git clone -q ~/.abra/recipes/discourse $LA/recipes/discourse
git -C $LA/recipes/discourse checkout -f -q -B main 53ba0910
git -C $LA/recipes/discourse remote set-url origin $LA/recipes/discourse
for sh in catalogue servers; do ln -s $(realpath ~/.abra/$sh) $LA/$sh; done
ABRA_DIR=$LA script -qec "abra recipe lint -n discourse" /dev/null   # -> R011 X "invalid reference format" x2
# vs the same flow at 0.8.1+3.5.0 -> R011 OK
```

**Proposed remedy (recipe PR #4):** remove the orphaned `sidekiq:` block from `compose.smtpauth.yml` (fold
its `DISCOURSE_SMTP_PASSWORD_FILE` env + `smtp_password` secret into the `app` service, since sidekiq is now
internal). Re-run discourse cold -> EXPECT R011 OK, level=5. Only the Adversary closes this, after re-test.

### [adversary] F-redfix-2 — live API key sat untracked **and un-gitignored** at the Builder clone's repo root (`config.json`) — one `git add -A` from being pushed to origin — **CLOSED @2026-07-08T23:26Z**

**CLOSED by Adversary cold re-test.** Builder remedied @`8cf08fd`: `config.json` added to the "local secrets /
env — never commit" block in `.gitignore` (line 7, with a comment naming the finding). My independent
verification, none of it taking the Builder's word:
1. **Attack replay from cold** — `git add -A` into a scratch `GIT_INDEX_FILE` seeded from HEAD: staged paths
   are `main.go` only; `config.json` **not staged**. `git check-ignore -v config.json` → `.gitignore:7`.
2. **Fix is on origin, not just local** — `git show origin/main:.gitignore` contains `config.json`. A *fresh
   clone from origin* + dropping the real `config.json` in → ignored ✅, `git add -A` does not stage it ✅.
   This matters: a local-only .gitignore edit would not protect the next clone.
3. **Full key never committed** — my original evidence used the 6-char prefix and is now **contaminated**: our
   own finding/inbox/journal text contains `tk_bhg`, so `-S'tk_bhg'` yields false positives. Re-tested against
   the *full 51-char value*: `git log --all -S"$KEY"` → **0 commits** in BOTH `cc-ci` and `cc-ci-adv`. Binary
   search on prefix length: the longest prefix ever committed anywhere is **6 of 51 chars**, in our own
   documentation — not a usable disclosure. No leak, past or latent.
4. **No non-git exposure** — dashboard is live (`https://ci.commoninternet.net/` → 200) but
   `/config.json` → **404** (also 404 on `dashboard.ci.…`); no tracked source reads it (the other
   `config.json` hits are `/root/.docker/config.json`, unrelated). Perms `-rw-r--r-- loops:users`.

**CORRECTION to my own finding (Builder was right, I was wrong).** I wrote "BOTH Builder clones". There is
only **one** repo: `/srv/cc-ci` is a symlink → `/srv/cc-ci-orch` (`ls -ld`), so `/srv/cc-ci/cc-ci` and
`/srv/cc-ci-orch/cc-ci` share `rev-parse --show-toplevel`, the same `.git` inode (3206558) and the same
`.gitignore` inode (3252849). My `cc-ci-adv` "pair" is the same illusion. A filesystem-wide sweep found
exactly one `config.json` inside any git repo, and it is now IGNORED. One fix, fully applied — not half.

**Residual, explicitly NOT closed by this:** the key is still on disk **unrotated** (`len=51`, `tk_bhg…`).
Gitignoring prevents a future commit; it cannot un-expose a value that leaked by another channel. Since the
full key provably never entered git and is not HTTP-reachable, git is not a reason to rotate. The Builder
correctly **escalated rotation to the operator rather than deciding it** — that judgement was right, and the
call remains the operator's.

---

<details><summary>Original report (as filed 2026-07-08T23:12Z)</summary>

### [adversary] F-redfix-2 — original text — **OPEN, NON-BLOCKING**

**Severity:** does NOT block phase `redfix` (out of scope of its Definition of Done — no VETO, DONE stands).
Latent secret-leak risk in the working environment; worth fixing before any future phase does a broad `git add`.

**What:** `config.json` (1128 B, mtime 2026-06-23T00:50Z) exists at the repo root of BOTH Builder clones —
`/srv/cc-ci/cc-ci` and `/srv/cc-ci-orch/cc-ci`. It holds a live-shaped inference credential at
`.provider.tinfoil.options.apiKey` (51 chars, prefix `tk_bhg…` — value not reproduced here). The file is
**untracked**, but `.gitignore` does **not** cover it: `.gitignore` lists `.testenv`, `*.key`, `*.pem`,
`runs/`, `.claude/` — no `config.json`. So `git check-ignore config.json` → miss.

Origin is a real pushed remote (`git.autonomic.zone/recipe-maintainers/cc-ci.git`, credentials embedded in
the remote URL). A single `git add -A` / `git add .` in either clone would stage and then push the key.

**Good news (verified, not assumed):** the key has never been committed —
`git log --all --oneline -S'tk_bhg'` → empty; `git log --all -- config.json` → empty; `git ls-files` has no
`config.json` at any path. So this is a *latent* risk, not an existing leak. The Adversary clones
(`/srv/cc-ci/cc-ci-adv`, `/srv/cc-ci-orch/cc-ci-adv`) do not carry the file at all.

**Repro:**
```
cd /srv/cc-ci/cc-ci && git status --porcelain config.json     # -> "?? config.json"
git check-ignore -v config.json; echo "exit=$?"               # -> exit=1 (NOT ignored)
git log --all --oneline -- config.json                        # -> empty (never committed)
```

**Proposed remedy (Builder — repo change, mine to file, not to make):** add `config.json` to `.gitignore`
under the existing "local secrets / env — never commit" block. Optionally rotate the Tinfoil key if it was
ever pasted into a log/transcript. I did **not** touch, move, or delete the file — it holds a live-looking
credential and is not mine to modify.

**Discovery:** independent break-it probe on my "no secrets in the repo / published logs / dashboard"
standing mandate, run after the phase closed. The Builder's journal @418ec57 independently noticed the same
file; I verified the exposure surface (gitignore miss + never-committed) from a cold start rather than
taking that note at face value. Only the Adversary closes this, after re-test.

</details>

### [adversary] F-redfix-3 — M2's discourse evidence shas (`9ff5e19`, `53ba0910`) no longer exist on the mirror; the fix content survives — **CLOSED @2026-07-08T23:24Z (non-blocking, no VETO)**

**Severity:** does NOT block phase `redfix` (DONE stands). Evidence-durability defect in the *record*, not in
the fix. Filed so a future auditor of redfix does not conclude "the discourse fix was withdrawn."

**What.** `STATUS-redfix.md` pins the discourse fix at `9ff5e19` (fix list) and `53ba0910` (WHERE refs) on
`recipe-maintainers/discourse` branch `discourse-official-image`. As of 2026-07-08 that branch heads at
`ede6399` and **neither sha resolves**: fetching all 17 `refs/heads/*` + `refs/pull/*/head` into one clone and
running `git cat-file -t` on each returns *not a valid object name*. The branch was force-pushed/rebased and
extended by **later** phases (`ede6399` = `refs/pull/5/head`; adds `discourse/postgres:pg18` + `POSTGRES_USER`
in `pg_backup.sh`). redfix's PR is also no longer "#4" — `refs/pull/4/head` is now `0c4539b7`.

**Why it is CLOSED rather than a VETO.** I re-verified the *content* the M2 PASS actually asserted, at the
current head: `compose.yml` → `image: discourse/discourse:3.5.3` (official-image migration) and
`compose.smtpauth.yml` → 0 `sidekiq` occurrences (the F-redfix-1 remedy). Both hold at `ede6399` and at
`0c4539b7`. The fix is present and re-verifiable; only the pointers rotted. M2 was correct when given.

**Repro.** `git clone https://git.autonomic.zone/recipe-maintainers/discourse && cd discourse && git fetch
origin 'refs/heads/*:refs/remotes/origin/*' 'refs/pull/*/head:refs/remotes/pr/*' && git cat-file -t 9ff5e19`
→ fatal. Then `git show origin/discourse-official-image:compose.yml | grep image:` → official image present.
(Note: `git fetch origin <sha>` and a `--filter=blob:none` clone both give false "absent" signals — use
reachability from all refs.)

**Lesson for future phases (no action required of the Builder now):** shared recipe branches get rewritten, so
a sha alone is not durable evidence. Record the *content assertion* (file → expected line) alongside the sha,
or push a tag. The other three redfix fixes pinned exactly (`4ca7f418`, `a0f2db88`, `4987ba91`), as did cc-ci
`redfix-m2-harness`@`07fc6d4a` — discourse drifted only because a later phase reused its branch.

### A-redfix-2 [adversary] — STATUS's "no code path redeploys it / recovery is manual" is REFUTED: the B-redfix-5 wedge **does** self-heal on the next unit activation, which makes it look intermittent

Severity **LOW-MED**, **no VETO, does not reopen the phase** (operator-facing description of a *deferred*
item, not a DoD item). Merge target `b5f2b10` unchanged. Filed because the stated mechanism is wrong and the
wrong mechanism points the operator at the wrong remediation.

**What STATUS-redfix.md (`68b51d5`, lines ~91-94) claims:**
> "**The wedge does not self-heal.** … A raise at `:514` or `:536` therefore escapes to `SystemExit` with
> live keycloak already undeployed by the preceding `abra.undeploy()` — **no code path redeploys it.
> Recovery is manual.**"

**The conclusion "does not self-heal" is right; the reason given is wrong.** There *is* a code path that
redeploys it, and it runs automatically.

**Repro (static, from any clone; no node access needed):**

    git show b5f2b10:runner/warm_reconcile.py | sed -n '468,479p'   # the fresh-deploy branch
    git show b5f2b10:runner/harness/abra.py   | sed -n '295,298p'   # undeploy does NOT remove the .env
    sed -n '38,45p' nix/modules/warm-keycloak.nix                   # oneshot, RemainAfterExit, no Restart=
    git grep -n "warm-keycloak" -- nix/ | grep -iE "timer|OnCalendar|Restart="   # -> empty

**Chain, each leg verified:**
1. `abra.undeploy()` runs only `abra app undeploy` — it does **not** remove `~/.abra/servers/default/<domain>.env`.
2. So on the next `reconcile("keycloak")`: `current_version(domain)` still resolves from `TYPE=` in that
   `.env`, while `is_deployed(domain)` is `False` (no docker services for the stack).
3. `warm_reconcile.py:471-479` therefore takes the **fresh-deploy branch**: `target = current or latest` →
   `deploy_version(...)` → `wait_healthy` → `write_last_good` → `return deployed-fresh:<target>`.
   **This branch never calls `warmsnap`**, so `_assert_slot_not_foreign` is never reached and the foreign
   meta cannot block it. Live keycloak comes back up.

**What actually prevents self-healing** is not the absence of a redeploy path but the absence of a
**re-trigger**: `warm-keycloak.service` is `Type=oneshot`, `RemainAfterExit=true`, `wantedBy=multi-user.target`,
with **no timer and no `Restart=`** (the module comment says it "converges every activation/boot"). Nothing
re-runs it between activations.

**Why this matters operationally — the failure is intermittent, not sticky.**
- The wedge persists only until the next **reboot or `nixos-rebuild switch`**, at which point the unit
  re-activates, hits the fresh-deploy branch and silently restores live keycloak. "Recovery is manual" will
  mislead an operator into hand-redeploying something that a routine rebuild already fixed.
- But the foreign `snapshot/meta.json` is **still in the slot**. So the box heals, then **re-wedges the next
  time an upgrade is actually due** (`current != latest` → upgrade path → `:514` `snapshot()` → guard raises
  → `abra.undeploy()` at `:512` has already run). Heal → re-wedge → heal: it will read as a *flake*.
- Therefore the correct remediation is **remove the foreign `snapshot/` from `/var/lib/ci-warm/keycloak/`**,
  not "redeploy keycloak". Redeploying treats the symptom and the wedge returns on the next due upgrade.

**Ask of the Builder:** correct the two sentences in STATUS. Suggested: *"The wedge does not self-heal between
activations — `warm-keycloak.service` is a `oneshot` with no timer and no `Restart=`. The next reboot or
`nixos-rebuild switch` re-activates it and the fresh-deploy branch (`:471-479`, which never calls `warmsnap`)
does restore live keycloak — but the foreign meta remains, so it re-wedges on the next due upgrade. Remediation
is to remove the foreign `snapshot/`, not to redeploy."* The **precondition itself is unchanged and still
correct**; only the mechanism/remediation sentences are wrong.

### A-redfix-3 [adversary] — STATUS's `main.go` "present in both clones ⇒ written outside git" evidence is void: `/srv/cc-ci` is a **symlink** to `/srv/cc-ci-orch`, so there is one file, not two

Severity **INFO**. No VETO, no gate impact, no DoD impact. The *conclusion* (untracked, not from this phase,
leave it alone) is fine; the supporting evidence is an artifact.

**What STATUS-redfix.md (`42bc4e4`, lines ~470-472) claims:**
> "`/srv/cc-ci-orch/cc-ci/main.go` **and** `/srv/cc-ci/cc-ci/main.go` — present in **both** clones, identical
> size, identical mtime … i.e. written by something outside git (**a git operation cannot leave the same
> untracked file in two clones**)."

**Repro:**

    ls -la /srv/                                 # cc-ci -> /srv/cc-ci-orch   (symlink)
    stat -c '%d:%i  %n' /srv/cc-ci-orch/cc-ci/main.go /srv/cc-ci/cc-ci/main.go
    # -> 2049:3254604 for BOTH; stat -c 'links=%h' -> links=1

Same device, same inode, link count 1: **one file, one clone, seen through a symlink.** The same holds for the
Adversary clones (`/srv/cc-ci-orch/cc-ci-adv` and `/srv/cc-ci/cc-ci-adv` are inode `3206945`). The "two clones"
observation carries **zero** information about the writer, and the identical size/mtime are tautological.

**Risk bounded (read-only):** contents are a 281-byte hello-world `net/http` listener; `sha256
bdbc3bf167cd20f30c00880005f4f994f17f3722660973c9c65c7bf33e81ffaf`; no `go.mod`; **`go` is not installed**;
**nothing is listening on `:8080`**; `git log --all -- main.go` is empty; not gitignored; unreferenced by any
tracked file. It cannot execute. Present in the **Builder's** clone only — **absent from both Adversary
clone paths**, which is consistent with a single stray write into one working tree.

**Ask of the Builder:** drop the "both clones ⇒ outside git" inference (keep the file, keep the flag). Agreed
on not deleting it: neither of us created it.