cc-ci/machine-docs/JOURNAL-redfix.md

# JOURNAL — phase `redfix`

## 2026-06-17T23:20Z — Bootstrap

Read phase plan + plan.md §6.1/§7/§9 + canon DECISIONS exceptions (lines ~1494–1552). Six
canon-sweep failures to investigate. Confirmed cc-ci access, no run in flight, sweep timer next
fires 2026-06-21 (3-day window), disk 38G free.

Isolation mechanism understood: `runner/nightly_sweep.run_on_tag` = `abra.recipe_checkout(r, tag)` +
`run_recipe_ci.py RECIPE=<r> CCCI_SKIP_FETCH=1` cold/full. I reproduce each failure by running ONE
recipe at a time with no concurrent load.

Starting canonical state notable: **mumble canonical IS present** (`1.0.0+v1.6.870-0`, written
20260617T180501Z — during today's nixenv sweep). The canon DECISIONS recorded mumble RED
(`test_handshake_completes_with_channel_presence`). A canonical only gets written on a GREEN cold run
on latest → mumble flipped green in a recent run. Strong early evidence for the operator's "mumble
passed before" → load flake hypothesis. Must confirm with a clean isolation re-run + check whether the
canon-sweep red was under concurrent load.

Next: start M1 investigation. Plan order (cheap/informative first): triage the existing sweep logs on
cc-ci to pin the EXACT assertion/error for each (mumble, mattermost-lts restore, gitea app.ini,
bluesky routing, discourse compose), then run isolation re-runs. discourse's recorded cause is an
UPSTREAM compose defect (`sidekiq.depends_on: discourse` while service is `app`) that FATAs before any
deploy — that's deterministic, not a load timeout, so it may not even need a long isolation run to
confirm; verify the compose at the latest tag directly first.

## 2026-06-17T23:40Z — M1: discourse isolation run — CANON ROOT-CAUSE WAS WRONG

Ran discourse ALONE on cc-ci (`recipe_checkout discourse 0.8.1+3.5.0` + `RECIPE=discourse
CCCI_SKIP_FETCH=1 cc-ci-run runner/run_recipe_ci.py`, log `/tmp/redfix-discourse.log`).

RESULT: **install PASS, upgrade FAIL, backup PASS, restore PASS, custom PASS** — the recipe deploys,
serves (200 /srv/status), backs up and restores cleanly. NOT a deploy timeout, NOT a 51-min wedge,
NOT a deploy FATA. The canon DECISIONS root-cause ("`abra app deploy` FATAs: service sidekiq depends
on undefined service discourse → invalid compose project") is **misattributed**: that string appears
ONLY from the non-fatal prepull `docker compose config --images` (rc=15, harness logs "skipping
(deploy will pull as usual)"). The real `abra app deploy` is a swarm `docker stack deploy`, which
ignores `depends_on` entirely → the stack converges (`UpdateStatus=completed`).

The ONLY failure is the cc-ci upgrade OVERLAY `tests/discourse/test_upgrade.py`:
- `test_head_runs_official_image_not_bitnamilegacy` — app image is `bitnamilegacy/discourse:3.5.0`;
  test demands `discourse/discourse:3.5.3` (official).
- `test_sidekiq_service_dropped_by_head` — services `['app','db','redis','sidekiq']`; test demands
  sidekiq dropped.

These `prevb`-phase overlay tests are PR-FAITHFULNESS assertions for a specific migration PR
(bitnamilegacy → official `discourse/discourse:3.5.3`, drop sidekiq). Verified that migration exists
in **NO upstream release tag and NOT in main** — `git show main:compose.yml` and every tag
(`0.1.0…0.8.1+3.5.0`) all use `bitnamilegacy/discourse:3.5.0` + sidekiq. So the overlay asserts a
state that doesn't exist anywhere upstream → deterministic RED whenever the sweep tests the latest
release tag. The head DID deploy (chaos-version label = head f87c612d+U, converged) — the test
expectation is simply wrong for the released recipe.

Note (M2 design): migrating discourse from the deprecated `bitnamilegacy` image to official
`discourse/discourse` is a MAJOR recipe rewrite (different fs layout, entrypoint, no `/opt/bitnami`
sidekiq run.sh) — not a 1-line image swap. So the overlay test's `discourse/discourse:3.5.3`
expectation may not be a realistic near-term recipe change. The bitnamilegacy deprecation is real
(bitnami sunset legacy images), so a migration is the right long-term direction, but the test as
written hard-codes a migration target absent upstream. Classification + fix approach to settle in M1
table / M2.

Classification: **stale/PR-specific cc-ci OVERLAY test mismatched to the canonical-sweep context**
(NOT a flake, NOT a load timeout, NOT a recipe-deploy defect, NOT warm-machinery). Teardown clean (no
discourse stack left). Evidence: `/tmp/redfix-discourse.log` on cc-ci; junit under
`/var/lib/cc-ci-runs/manual/junit/upgrade__cc-ci__test_upgrade.xml`.

## 2026-06-18T00:05Z — M1: mattermost-lts isolation run — DETERMINISTIC restore failure (recipe defect)

Ran mattermost-lts ALONE (tag 2.1.9+10.11.15, log /tmp/redfix-mattermost-lts.log).
RESULT: **install/upgrade/backup/custom PASS, restore FAIL** — identical to the canon failure:
`tests/mattermost-lts/test_restore.py::test_restore_returns_state` → `relation "ci_marker" does not
exist` after restore. So it is **deterministic in isolation, NOT a loaded-node race** (canon framing
was wrong). The marker logic is sound (postgres table seeded pre-backup, dropped pre-restore, asserted
post-restore — same pattern immich uses and PASSES).

ROOT CAUSE (recipe backup/restore labels). Compared mattermost-lts vs immich (immich passes the
IDENTICAL test):
- immich `database` svc: `backupbot.backup.pre-hook: /pg_backup.sh backup`,
  `backupbot.backup.volumes.postgres.path: backup.sql` (backs up ONLY the dump file), and
  **`backupbot.restore.post-hook: /pg_backup.sh restore`** (replays the dump on restore). → round-trips.
- mattermost-lts `postgres` svc: `pre-hook: pg_dump > /var/lib/postgresql/data/postgres-backup.sql`,
  `backup.path: /var/lib/postgresql/data/` (backs up the WHOLE live/hot PGDATA dir + the dump),
  `post-hook: rm .../postgres-backup.sql`, and **NO `backupbot.restore.post-hook`**. So on restore,
  abra restores the files but NOTHING replays the dump, and a hot-copied live PGDATA over a running
  postgres does not reload → `ci_marker` lost. Restore log confirms `Restoring Snapshot b0495d36 at /`
  with no post-hook reimport.

Classification: **GENUINE RECIPE DEFECT at latest** (postgres backup/restore does not round-trip —
missing restore post-hook + backs up hot PGDATA instead of dump-only). NOT a flake, NOT cc-ci test
weakening (test is correct & unmodified; immich proves the pattern works). Fix (M2) = recipe PR
adopting the immich-style postgres backup/restore (a `/pg_backup.sh`-style dump + restore post-hook).
Teardown clean (no matt stack). Evidence: /tmp/redfix-mattermost-lts.log; junit
restore__cc-ci__test_restore.xml.

Tooling note: my background "waiter" loop `while pgrep -f run_recipe_ci.py` self-matched (its own
cmdline contains the string) → never exited, falsely showed a run active. Use `pgrep -f
"[r]un_recipe_ci.py"` or match the python invocation. Killed the stuck waiters; node confirmed free.

## 2026-06-18T00:18Z — M1: mumble isolation run — GREEN (flake confirmed)

Ran mumble ALONE (tag 1.0.0+v1.6.870-0, log /tmp/redfix-mumble.log). RESULT: **ALL tiers PASS**
(install/upgrade/backup/restore/custom), including `custom/test_protocol_handshake.py::
test_handshake_completes_with_channel_presence` PASSED. No orphan stacks. The canon sweep recorded
this RED (`test_handshake…` failed under concurrent sweep load); it is GREEN here in isolation, and
its canonical was already written green TODAY (1.0.0+v1.6.870-0 @20260617T180501Z) under the lighter
nixenv sweep. → **load/timing FLAKE** on the control-channel handshake, NOT a recipe defect.

The handshake test already retries (`retry_handshake(attempts=12, interval=5.0)` = 60s). So the flake
is the voice server not completing the TLS+ServerSync handshake within ~60s under heavy concurrent
node load (deploy contention). M2 fix = harness stabilization (stronger readiness gate before the
custom tier / longer-or-smarter retry / serialize), based on the load failure mode. Classification:
**FLAKE (load/concurrency)** → harness stabilization.

Reproducibility: 1 green isolation run here + canonical green today + documented red under canon load.
Will do 1–2 more isolation repeats before the M1 claim to firm "reproducibly green in isolation."

## 2026-06-18T00:45Z — M1: bluesky-pds isolation run — 000 REPRODUCES; root cause = `app` DNS collision on shared proxy

Ran bluesky-pds ALONE (tag 0.3.0+v0.4.219, log /tmp/redfix-bluesky-pds.log). Cold lifecycle GREEN
(install/backup/restore/custom pass; upgrade EXPECTED_NA per recipe_meta — moving pds:0.4 tag). Then
WC5 promote-on-green-cold FAILED exactly as canon: `warm-bluesky-pds.ci.commoninternet.net: not
healthy over HTTPS /xrpc/_health (last status 0)`. So **the 000 reproduces deterministically in
isolation — NOT a sweep-load/ACME-rate-limit flake** (my first hypothesis, refuted).

LIVE DIAGNOSIS (stack left deployed by the failed promote; probed before teardown):
- app service 1/1, healthy: `docker exec app wget localhost:3000/xrpc/_health` → `{"version":"0.4.219"}`;
  app listens on `:::3000`; no restarts. So the PDS itself is fine.
- HTTPS to warm domain → 000. caddy logs flood:
  `tls "failed to get permission for on-demand certificate" domain=warm-bluesky-pds…
   error=… Get "http://app:3000/tls-check?domain=…": dial tcp 10.10.0.X:3000: connect: connection refused`
  (X varies: .2 .4 .5 .6 .8 .9 .10 .12).
- bluesky uses caddy **on-demand TLS** (Caddyfile: `on_demand_tls { ask http://app:3000/tls-check }`,
  `tls { on_demand }`, `reverse_proxy app:3000`). caddy must reach app:3000/tls-check to be GRANTED a
  cert before serving TLS. It can't → no cert → TLS handshake fails → 000.
- WHY can't caddy reach app: **service-name `app` collision on the shared `proxy` overlay.**
  - app is on `warm-bluesky-pds…_internal` ONLY (IP 10.0.3.3). caddy is on `proxy` (10.10.50.223) +
    `…_internal` (10.0.3.6).
  - `docker exec caddy getent hosts app` → returns ONLY proxy IPs (8/8 tries: 10.10.0.4/.5/.6/.10/.12),
    **NEVER the internal 10.0.3.3.** The proxy-net `app` alias shadows bluesky's own internal app.
  - `docker network inspect proxy` shows EVERY stack aliases its main service `app`:
    `drone…_app=10.10.0.2`, `traefik…_app=10.10.0.5`, `warm-keycloak…_app=10.10.0.9`,
    `ccci-reports/bridge/dashboard_app`, … — exactly the IPs caddy hits. None listens a PDS on 3000 →
    connection refused.
  So caddy resolves bare `app` to OTHER stacks' app endpoints on the shared proxy, never its own PDS.

WHY cold passes / warm fails: cold's health window is long (HTTP_TIMEOUT=600) and on first success
caddy CACHES the issued cert; the promote's shorter health window doesn't give caddy a chance to ever
resolve correctly (and here it provably never resolves to 10.0.3.3 at all). The collision is the root
cause; the promote machinery is CORRECT (it refused to write a canonical for an unhealthy 000 — no
canonical.json written, verified).

Classification: **genuine ROUTING/recipe defect — caddy↔app cross-stack `app`-alias collision on the
shared proxy net**, deterministic, reproducible in isolation. NOT a flake; NOT a promote-machinery bug.
Fix approach (M2): recipe PR giving the PDS service a UNIQUE name/alias (e.g. rename `app`→`pds`) so
caddy's `reverse_proxy`/`tls-check` resolve only bluesky's own internal service (no shared-proxy `app`
collision). (Alternatively a caddy-side internal-only resolution; renaming is cleanest.) Will confirm
the exact fix in M2 + verify the warm domain then serves 200.

Cleanup: removed orphaned warm-bluesky-pds stack + its volumes/secrets (promote had left it deployed;
no canonical written). Node clean.

## 2026-06-18T01:05Z — M1: keycloak — warm-domain namespace collision (harness), classification complete

keycloak was de-enrolled (WARM_CANONICAL=False) because its data-warm canonical domain would collide
with the LIVE-warm OIDC provider. Verified the collision STRUCTURALLY (code, no run needed):
- `canonical.canonical_domain(r)` → `warm.stable_domain(r)` → `f"warm-{r}.ci.commoninternet.net"`
  (runner/harness/canonical.py:42-44, warm.py:44-48).
- `warm.WARM_DOMAINS["keycloak"] = "warm-keycloak.ci.commoninternet.net"` (warm.py:27-29) — the
  always-on shared OIDC provider lasuite-*/drone consume for SSO; kept current by roll_warm_infra.
- So `canonical_domain("keycloak") == WARM_DOMAINS["keycloak"]` EXACTLY. Enrolling keycloak as a
  data-warm canonical → the sweep's promote deploy/teardown at warm-keycloak collides with the live
  provider. Confirmed live keycloak healthy (200 /realms/master) — I did not disturb it.

The collision is unique to keycloak: it is the ONLY recipe that is both a live-warm provider (in
WARM_DOMAINS) AND would want a canonical. No collision-free canonical namespace exists today.

Classification: **HARNESS defect — warm canonical domain namespace can collide with a live-warm
provider.** NOT a recipe/flake. Fix approach (M2): make `canonical_domain(r)` collision-free when `r`
is a live-warm provider — e.g. `warm-canon-<r>` (or unconditionally) so the canonical deploy gets a
distinct domain → distinct stack → cannot touch the live `warm-keycloak`. Then set keycloak
WARM_CANONICAL=True and verify it promotes at the collision-free domain WITHOUT disrupting live
keycloak. Minimal blast radius: special-case only providers in WARM_DOMAINS (the 15 other canonicals
keep `warm-<r>`); confirm in M2.

## 2026-06-18T01:05Z — M1: gitea first advance attempt hit a LEFTOVER confound (not the real crash)

First gitea cold@3.6.0 run: cold lifecycle (install/upgrade/backup/restore/custom) ALL PASS; promote
advance FAILED with `FATA warm-gitea.ci.commoninternet.net is already deployed` — NOT the app.ini
crash. Cause: warm-gitea was left DEPLOYED at 3.5.3 by the nixenv-phase sweep (registry said
status=idle but the stack was actually running — a state inconsistency). The advance does `abra app
deploy warm-gitea` assuming the canonical is idle/undeployed; finding it deployed, abra FATAs. This is
the same GREEN-BUT-PROMOTE-FAILED the nixenv phase saw. To reproduce the REAL app.ini issue I undeployed
warm-gitea (docker stack rm; retained data+config volumes → proper idle state) and re-ran gitea
cold@3.6.0 (gitea2). Result pending. NOTE: the "already deployed" promote-failure-when-left-deployed
may be a secondary promote-machinery robustness gap (advance should undeploy-or-chaos an
already-deployed canonical) — will assess after confirming the primary app.ini crash.

## 2026-06-18T00:14Z — M1: gitea warm advance — app.ini read-only JWT crash CONFIRMED (recipe defect)

After restoring warm-gitea to proper idle state (undeployed, 3.5.3 data+config volumes retained),
re-ran gitea cold@3.6.0 (gitea2, log /tmp/redfix-gitea2.log). Cold lifecycle ALL PASS
(install/upgrade/backup/restore/custom — incl. the cold FRESH 3.5.3→3.6.0 upgrade tier). WC5 promote
advance then crash-loops. Live container logs (warm-gitea_..._app, repeated Failed/exit 1):

  modules/setting/setting.go:105:LoadCommonSettings() [F] Unable to load settings from config:
  error saving JWT Secret for custom config: failed to save "/etc/gitea/app.ini":
  open /etc/gitea/app.ini: read-only file system

EXACTLY the canon-documented crash. Mechanism: the recipe mounts app.ini as a docker `config`
(read-only by design) at /etc/gitea/app.ini (compose `configs: - source: app_ini target:
/etc/gitea/app.ini`, app.ini.tmpl). gitea 1.24.2 (3.6.0), on the warm REATTACH of the retained
3.5.3 config volume, decides to (re)generate+SAVE a JWT secret to app.ini → read-only fs → FATA at
config-load, BEFORE any DB migration (so the 3.5.3 data volume stays intact — confirmed canon).

Why cold passes but warm crashes: the cold fresh deploy + cold chaos-upgrade use freshly-generated
secrets consistent with a freshly-initialized config, so gitea never needs to rewrite app.ini. The
warm advance reattaches an OLDER retained config-volume state (seeded under 3.5.3) against the new
run's secrets/3.6.0 binary → gitea reconciles by trying to persist a JWT secret → read-only crash.

Classification: **genuine RECIPE defect** (gitea 3.6.0/1.24.2 + read-only app.ini docker-config mount
on the warm-reattach advance), deterministic, reproduced first-hand. NOT a flake, NOT promote
machinery. Fix approach (M2): recipe PR making app.ini writable on the advance path — e.g. render the
config into the WRITABLE `config:/etc/gitea` volume via an entrypoint (not a read-only docker config),
OR ensure the persisted secrets are accepted without rewrite. (Secondary harness option: canonical
advance falls back to clean re-deploy when in-place config rewrite is impossible — but that loses the
reattach data-warm property; recipe fix preferred.) Ties to LFS PR #1 (app.ini secret handling).

ACTION NEEDED after run exits: warm-gitea is left crash-looping at 3.6.0 → restore it to 3.5.3
(redeploy the known-good canonical version) so the canonical is healthy again. Data volume intact.

## 2026-06-18T00:25Z — M1 CLAIMED (6/6 investigated, isolated, classified)

mumble repeat #2 (mumble2): ALL tiers green again incl. handshake; canonical re-promoted green
(ts 20260618T001730Z). So mumble = 2× reproducibly green in isolation → load/timing FLAKE confirmed.

All six classified with first-hand isolation evidence (or code proof for keycloak). Two canon
root-causes were CORRECTED by isolation: discourse (not a timeout/deploy-FATA — it's a stale cc-ci
overlay test asserting an unreleased migration) and mattermost-lts (not a loaded-node race — a
deterministic recipe restore defect: missing `backupbot.restore.post-hook`). bluesky's 000 is NOT a
load/rate-limit flake (my initial hypothesis) but a deterministic caddy↔app `app`-alias DNS collision
on the shared proxy. gitea app.ini read-only JWT crash reproduced first-hand. keycloak collision proven
structurally in code.

Node clean: warm-gitea idle@3.5.3 (volumes retained), orphaned warm-bluesky removed, only live
warm-keycloak up (healthy 200). Claiming M1; will start M2 fix design while awaiting the Adversary
verdict (keep an unblocked item in hand).

## 2026-06-18T00:25Z — M2 prep (gated on M1 PASS): bluesky fix refinement

While parked at the M1 gate (no node deploys — Adversary cold-verifying), refined the bluesky fix:
cc-ci's bluesky tests probe via HTTP (/xrpc/_health), but the GENERIC harness defaults to
`service="app"` (deployed_identity/_app_container). So RENAMING the recipe's `app` service → `pds`
could break generic harness assumptions. Cleaner fix: keep the service named `app` but give it a
UNIQUE network ALIAS on the internal net (e.g. `aliases: [pds-internal]`) and point caddy at
`pds-internal:3000` (reverse_proxy + on_demand_tls ask). A unique alias has no collision on the shared
proxy (only the bare `app` alias collides), and the service name stays `app` → zero cc-ci-side
breakage. Will validate this exact approach in M2 after M1 PASS.

## 2026-06-18T01:21Z — M1 PASS; starting M2

Adversary M1 verdict: **PASS** @01:18Z — all 6 classifications cold-verified CORRECT by its OWN
isolation re-runs (discourse/mattermost/mumble/bluesky/gitea) + code-verify (keycloak). No VETO.
"Builder cleared to proceed to M2." Two canon root-causes corrected and confirmed (discourse: not a
timeout, stale overlay; mattermost: not a load race, recipe defect). bluesky reclassification (recipe,
not warm-machinery) confirmed against the plan's prior.

Starting M2. Plan: recipe PRs (mattermost-lts, bluesky-pds, gitea) via the recipe mirror+PR flow
(`!testme`-verified, never merge); harness fixes (keycloak collision-free canonical_domain + enroll;
mumble handshake stabilization) on a cc-ci branch; discourse overlay-scope decision. Node now mine
(Adversary done). Will examine the recipe-create-pr flow first, then execute one fix at a time.

## 2026-06-18T01:25Z — M2 recon: prior-phase fix PRs already exist for discourse + mattermost

Surveyed open PRs on all 6 mirrors before doing redundant work:
- **discourse #4** `discourse-official-image` ("switch to official discourse/discourse"): created
  2026-06-16 by autonomic-bot; **!testme PASSED twice**, latest @53ba0910 today 16:36Z (run #849) ✅.
  This migrates off deprecated bitnamilegacy → official image + drops sidekiq = EXACTLY what the
  upgrade overlay asserts. So the overlay test was correctly demanding the migration; PR #4 IS the
  discourse fix and is already !testme-green. (Reframes M1 "stale test": the test is right; the
  release tag predates the migration; the fix is the migration PR, not weakening the test.)
- **mattermost-lts #1** `ci/pg-restore` ("reimport the postgres dump on restore"): correct
  immich-pattern fix — pg_backup.sh (backup pg_dump|gzip; restore: terminate conns + DROP DATABASE
  WITH FORCE + createdb + reimport) + dump-only `backup.volumes.postgres_data.path: backup.sql` +
  `restore.post-hook: /pg_backup.sh restore`. Created 2026-05-30; needs a fresh !testme to confirm
  green NOW. (Also PR #2 upgrade-2.1.11 overlaps — adds restore hook + version bump; #1 is the focused
  fix.)
- mumble #1 = "cfold sweep probe" (not the fix — mumble is a harness flake, no recipe PR needed).
- bluesky #3 = version bump (not the routing fix — need a NEW PR for the app-alias collision).
- gitea, keycloak = no open PRs (gitea LFS #1 closed; keycloak is a harness fix).

M2 plan refined: VERIFY discourse #4 (re-!testme fresh) + mattermost #1 (!testme); CREATE recipe PRs
for bluesky (unique alias) + gitea (app.ini writable); HARNESS fixes for mumble (handshake stab) +
keycloak (collision-free canonical_domain + enroll). Starting with mattermost #1 !testme.

## 2026-06-18T01:30Z — M2: mattermost-lts FIXED (verified) + discourse already green + bluesky PR created

- **mattermost-lts**: !testme on PR #1 `ci/pg-restore` (@4ca7f418) → run #901 ALL tiers green
  (install/upgrade/backup/restore/custom, every junit failures=0 skipped=0). The M1-failing
  `restore__cc-ci__test_restore.py::test_restore_returns_state` now PASSES — the pg_backup.sh restore
  post-hook (terminate conns + DROP DATABASE WITH FORCE + createdb + reimport dump) round-trips
  postgres state. **FIXED + verified.** (Nothing merged — operator merges.)
- **discourse**: PR #4 `discourse-official-image` already !testme-green @53ba0910 (run #849, today
  16:36Z) — the official-image migration makes the upgrade overlay pass. Will re-verify fresh for
  current evidence before the M2 claim.
- **bluesky-pds**: created mirror PR #4 `ci/warm-routing-alias` (unique `pds` alias on internal +
  caddy reverse_proxy/ask → pds:3000; service stays `app`). compose validated (`docker compose config`
  rc=0). VERIFICATION NOTE: bluesky's 000 is warm-promote-only (cold path always green), so !testme
  (cold) won't reproduce/verify it — I'll verify by running the FIXED recipe through the promote path
  (cold-on-latest with the fix checked out) → warm-bluesky-pds should serve 200 (vs M1's 000), then
  tear down the phantom canonical.

Remaining M2: bluesky promote-verify, gitea recipe PR (app.ini writable), keycloak harness
(collision-free canonical_domain + enroll), mumble harness (handshake stabilization).

## 2026-06-18T02:10Z — M2 bluesky: alias fix blocked by abra; pivoting to service RENAME

Verified the bluesky `pds` network-alias fix end-to-end and found a blocker:
- `docker stack deploy` HONORS compose network aliases (throwaway test: app got `Aliases:["pds","app"]`).
- `docker compose config` PRESERVES the alias in its render.
- BUT the harness/abra promote deploy produced an app service with `Aliases:["app"]` only — the `pds`
  alias was DROPPED. The fixed Caddyfile (pds:3000) DID deploy (same per-run tree), so abra read my
  recipe tree; by elimination, **abra's own compose→swarm translation drops service network aliases**
  (it's not docker, not the tree). Also confirmed: the bluesky promote is a non-chaos pinned deploy.
  (Two stale-config gotchas also hit + fixed: docker configs are immutable+versioned — a stale
  `warm-bluesky..._caddyfile_v1` was reused until I removed it; lesson for gitea = bump config versions.)

→ Pivot to the ROBUST fix: RENAME the PDS service `app`→`pds`. Docker auto-adds the service short-name
as a network alias (abra can't drop that — the deployed `app` proved the service-name alias is always
applied), so caddy's `reverse_proxy pds:3000` resolves THIS stack's PDS (unique on internal; no `pds`
on the shared proxy). Coupled cc-ci change: 2 `exec_in_app(...)` calls default `service="app"`
(`tests/bluesky-pds/_p4.py:40`, `custom/test_account_and_post.py:49`) → must become `service="pds"`
(NOT a weakening — same assertion, correct service). The warm-routing PROOF (warm-bluesky-pds→200) is
the promote path (custom exec tests not involved); cold !testme-green needs the cc-ci ref update.

Need to determine how cc-ci-side code reaches a !testme run (also required for keycloak + mumble
harness fixes) — investigating CCCI_REPO/Drone checkout next.

## 2026-06-18T02:15Z — cc-ci-side change verification mechanism (for bluesky-rename/keycloak/mumble)

The Drone !testme build clones cc-ci at main HEAD; the manual runner runs from CCCI_REPO (default
/etc/cc-ci). To verify a cc-ci-side change WITHOUT pushing main or disturbing /etc/cc-ci (shared with
Adversary): push the change to a cc-ci BRANCH, clone/checkout that branch to a temp dir on cc-ci, and
run `cd <tmp> && CCCI_REPO=<tmp> cc-ci-run runner/run_recipe_ci.py RECIPE=... CCCI_SKIP_FETCH=1`
(cc-ci-run is the deployed nix env; runner/ + tests/ come from my branch checkout). Restores cleanly.

bluesky-rename coupling: the warm-promote only fires on a FULLY-GREEN cold run, and bluesky's custom
tier exec_in_app defaults to service="app". So renaming app→pds REQUIRES the cc-ci exec-ref update
(service="pds") deployed via the temp-checkout for the cold run to go green and the promote to fire.
So: (1) recipe rename PR, (2) cc-ci branch with exec-ref update, (3) verify via temp-checkout run ->
cold green -> promote -> warm-bluesky-pds 200.

## M2 progress snapshot (2026-06-18T02:15Z)
- mattermost-lts: DONE (PR #1 ci/pg-restore, !testme run #901 all-green incl restore). 
- discourse: DONE (PR #4 discourse-official-image, !testme run #849 green; re-verify fresh for claim).
- bluesky-pds: PR #4 (alias) -> superseding with service RENAME app->pds + cc-ci exec-ref update; verify on promote path.
- gitea: fix READY locally (/tmp/redfix-gitea: app.ini->staging + docker-setup seed-once + DOCKER_SETUP_SH_VERSION v2); needs PR push + warm-advance verify.
- keycloak: harness fix (canonical_domain collision-free for WARM_DOMAINS recipes + enroll) NOT STARTED.
- mumble: harness fix (handshake readiness/retry stabilization) NOT STARTED.

## 2026-06-18T02:45Z — M2 progress: gitea PR + harness branch pushed; bluesky pivoted to rename

- **gitea**: opened recipe PR #2 `ci/app-ini-writable` (app.ini->staging + docker-setup seed-once +
  DOCKER_SETUP_SH_VERSION v2). Advance-path verification RUNNING (fixed 3.6.0 reattach to idle 3.5.3
  canonical; expect no app.ini crash + promote). cold lifecycle green so far (install + cold upgrade
  converged).
- **bluesky**: PR #4 updated alias->RENAME service app->pds (abra drops aliases). 3-line recipe diff,
  validates. Coupled cc-ci exec-ref change on branch.
- **cc-ci harness branch `redfix-m2-harness`** pushed (3 commits): keycloak (collision-free
  canonical_domain + WARM_CANONICAL=True), mumble (handshake budget 60s->180s), bluesky-pds
  (exec_in_app service=pds). Verified via temp-checkout runs (CCCI_REPO=<branch checkout>).
- Verification sequencing (node is single, serial): gitea advance (running) -> bluesky rename promote
  (needs branch exec-refs) -> keycloak canonical at warm-canon-keycloak (needs branch) -> mumble.
  NOTE: mumble "green under load" is hard to reproduce deterministically; plan = show branch run still
  green + reason about the budget (or construct concurrent load).

## 2026-06-18T03:00Z — M2 gitea fix v1 (seed) BROKE the transition — needs rework

gitea advance verification (fixed 3.6.0): install tier PASSED FULLY (fresh 3.6.0 + my fix: API 200,
admin auth OK — so the seed works for a FRESH deploy), but upgrade/backup/restore/custom ALL FAILED:
`READY_PROBE not ready: /api/v1/version (last status 404) within 600s` after the 3.5.3->3.6.0 chaos
redeploy → gitea came up in INSTALL-WIZARD mode (serves 200 but no API/admin = no valid app.ini).
The LFS custom test's repo-create also 404'd (same wizard-mode cause).

So my seed-once fix is fine for fresh install but FAILS the 3.5.3->3.6.0 transition — exactly the path
the canon fix needs. Likely cause: on the chaos redeploy from a 3.5.3 stack (docker_setup_sh_v1, no
seed) the docker-setup config didn't update to my v2 (seed) while compose moved app.ini to the staging
path → /etc/gitea/app.ini empty → wizard. (To confirm: reproduce + inspect the post-redeploy container
— is docker_setup_sh_v2 mounted? does /etc/gitea/app.ini exist? gitea log.) Reverted the fix from
cc-ci's gitea clone; warm-gitea intact (idle 3.5.3, promote didn't fire on the red cold run). gitea
recipe PR #2 stands but the fix needs a rework (likely: a more robust seed that runs regardless of
config version, OR provide a 1.24-valid oauth2 JWT secret so gitea never rewrites app.ini — investigate
WHY 1.24 regenerates it). Deferring gitea; proceeding to bluesky-rename / keycloak / mumble verifies.

## 2026-06-18T03:30Z — M2 bluesky verification BLOCKED by abra non-chaos tag-revert; keycloak/mumble next

Root cause of the bluesky rename verify failure: the deployed service was `..._app` (not `pds`).
`run_recipe_ci` CCCI_SKIP_FETCH copies my renamed clone to the per-run tree, BUT abra's NON-CHAOS
pinned deploy (bluesky's tag 0.3.0+v0.4.219 is ANNOTATED) does `git checkout <tag>` in the per-run
tree, REVERTING my rename to the tag's `app:`. So the renamed recipe never deployed; the branch
harness then execs `service=pds` -> "no running container <stack>_pds" -> backup/restore/custom red.
(This also re-explains the earlier "abra dropped the alias" — it was the same tag-revert, not a drop.)
gitea's tag is lightweight -> deploy_app uses chaos -> my gitea fix DID deploy (install passed); its
failure is a real transition issue, not a revert.

IMPLICATION: verifying a RECIPE fix (bluesky, gitea) via CCCI_SKIP_FETCH needs a CHAOS deploy (uses the
checkout, not the tag). HARNESS fixes (keycloak canonical_domain, mumble retry) are runner/test code
from the branch checkout — NO tag-revert — so they verify cleanly. Doing keycloak + mumble next.
For bluesky: force chaos (deploy_app does chaos when has_ccci_overlay) OR reconsider a cc-ci-side
overlay fix (alias + caddyfile override) — both verifiable; recipe PR #4 (rename) stays as the ideal
upstream fix. gitea: rework + reproduce-with-inspection.

## 2026-06-18T03:40Z — M2 keycloak FIXED + VERIFIED (collision-free canonical)

Ran keycloak cold-on-latest from branch checkout /tmp/cc-ci-m2run (harness fix: canonical_domain ->
warm-canon-keycloak for WARM_DOMAINS recipes; WARM_CANONICAL=True). RESULT: all cold tiers PASS
(install/upgrade/backup/restore/custom), and WC5 promote SUCCEEDED:
  canonical keycloak @ 10.8.0+26.6.3, domain="warm-canon-keycloak.ci.commoninternet.net", idle, volume retained.
- Promoted at the COLLISION-FREE domain warm-canon-keycloak (not warm-keycloak). ✓
- Live warm-keycloak (shared OIDC provider) = 200 THROUGHOUT — undisturbed. ✓
- warm-canon-keycloak = 404 now = CORRECT idle state (data-warm canonical undeployed, volume kept).
So keycloak is now a full data-warm canonical with zero risk to the live SSO. **FIXED + verified.**
3/6 verified: mattermost-lts, discourse, keycloak. Doing mumble next (harness, tractable).

## 2026-06-18T03:50Z — M2 mumble VERIFIED (stabilization); 4/6 done

Ran mumble from branch checkout (handshake budget attempts=36/180s). ALL tiers PASS incl
test_handshake_completes_with_channel_presence; promote succeeded (canonical 1.0.0+v1.6.870-0 idle).
The longer budget is active + non-regressing. NOTE: mumble is green in isolation regardless of budget
(the 60s sufficed in isolation); the budget matters UNDER LOAD, which is hard to reproduce
deterministically — so this verifies the stabilization is applied + sound + non-weakening, not a literal
load-flake repro. (M1 already established green-isolation/red-under-canon-load; the fix gives the
handshake 3x the readiness window.) **Stabilization fix verified.** 4/6: mattermost, discourse,
keycloak, mumble. Remaining: bluesky (force-chaos verify of the rename), gitea (rework).

## 2026-06-18T03:52Z — M2 bluesky force-chaos verification approach

bluesky's rename can't deploy via the normal path (annotated tag -> non-chaos -> abra checks out the
tag, reverting the rename). In PRODUCTION post-merge the new tag would carry the rename (non-chaos
deploys it fine). For PRE-merge verification I force chaos via a temporary tests/bluesky-pds/
compose.ccci.yml scaffold on the branch (has_ccci_overlay -> deploy_app uses chaos -> deploys my
renamed checkout). Then cold goes green (service pds + branch exec-refs) and the promote deploys the
renamed recipe at warm-bluesky-pds via chaos -> caddy resolves the unique `pds` -> expect 200 (vs M1
000). The overlay is a verification scaffold (NOT part of recipe PR #4); removed after.

## 2026-06-18T04:05Z — M2 bluesky verification: STRUCTURAL blocker (pre-merge warm-promote)

bluesky rename verification keeps deploying the TAG's `app:` (not my rename), even with: tag moved to
the rename commit AND a force-chaos overlay. Root: the warm-promote/cold-on-latest path resolves the
recipe at the UPSTREAM annotated tag (deploy_app recipe_checkout(tag) reverts unmerged content; the
chaos+overlay path STILL recipe_checkout's the pinned version). Unlike gitea (lightweight tag -> the
upgrade-tier chaos_redeploy uses the CHECKOUT, so the gitea fix deployed), bluesky has NO upgrade tier
(EXPECTED_NA) -> no chaos_redeploy path -> the rename never deploys on the promote path.

CONSEQUENCE: an unmerged RECIPE fix whose failure is WARM-PROMOTE-ONLY (bluesky 000) cannot be
end-to-end-verified via the standard harness pre-merge. mattermost/discourse were verifiable because
their failures are COLD tiers (restore/upgrade-overlay) reachable by !testme on the PR head.

bluesky fix correctness is nonetheless ESTABLISHED by: (1) M1 root cause (Adversary-confirmed): bare
`app` collides on the shared proxy; (2) docker test (proven): a unique service name/alias resolves to
the local service (no collision). Renaming app->pds (PR #4) gives a unique name -> caddy resolves THIS
PDS -> cert issued -> 200. End-to-end warm-200 needs either a DIRECT abra chaos deploy at
warm-bluesky-pds (manual app+secrets+PLC-key setup; next iteration) or operator post-merge verify.
Restored the bluesky tag; node clean; warm-keycloak 200.

## M2 STATUS (2026-06-18T04:05Z) — 4/6 verified
- mattermost-lts: VERIFIED (PR #1 ci/pg-restore, !testme run #901 all-green incl restore).
- discourse: VERIFIED (PR #4 discourse-official-image, !testme run #849 green).
- keycloak: VERIFIED (branch redfix-m2-harness; canonical promotes at warm-canon-keycloak, live warm-keycloak undisturbed 200).
- mumble: VERIFIED-stabilization (branch; green + budget 180s active; load-flake not deterministically reproducible).
- bluesky-pds: fix correct (PR #4 rename) + mechanically proven; end-to-end warm verify structurally blocked pre-merge -> direct-deploy or operator post-merge.
- gitea: PR #2 seed fix BROKE 3.5.3->3.6.0 transition (wizard mode); testable via chaos; NEEDS REWORK (reproduce+inspect).
NOT claiming M2 — bluesky end-to-end + gitea rework outstanding.

## 2026-06-18T05:53Z — M2 gitea VERIFIED (v3 seed) + bluesky VERIFIED (${STACK_NAME}_app); 6/6

**gitea — rework was already done (v3, a0f2db8) but unverified; verified it.** The clone's HEAD
a0f2db8 ("fix v2 -s seed, v3") already addressed the v1 wizard-mode bug: docker-setup seeds app.ini
into the writable /etc/gitea volume `if [ ! -s /etc/gitea/app.ini ]` (seed-on-EMPTY, not -f
seed-on-missing — a 3.5.3-old-recipe canonical leaves a 0-byte app.ini placeholder in the config
volume, which -f wrongly treats as present). Also bumps DOCKER_SETUP_SH_VERSION v1->v3 (config names
are immutable; forces swarm to re-mount the new docker-setup) + app.ini config target ->
/etc/gitea/app.ini.init (staging). Pushed v3 to PR #2 (force-replaced the broken v1 d4145266).

VERIFICATION (direct chaos-deploy onto the REAL idle 3.5.3 canonical volumes; /tmp/redfix-gitea-m2-directproof.log):
reattached the retained config volume (0-byte app.ini = genuine pre-fix M1 state) with the v3 recipe.
Result: app.ini seeded 0->1862 bytes, INSTALL_LOCK=true (not wizard), service 1/1, /api/v1/version
-> 200 {"version":"1.24.2"}, /api/healthz 200, retained 3.5.3 data adopted (data dirs dated
2026-06-17T08:39 = canonical seed time, not fresh), **0 read-only-app.ini crashes** (M1 crashed here).

WHY NOT the harness WC5 promote: it is STRUCTURALLY merge-gated. run_recipe_ci.py:373 force-fetches
`refs/tags/*` from upstream even under CCCI_SKIP_FETCH, and abra itself force-fetches tags on deploy
(abra.py:135 documents this) — so a LOCAL tag-move to the fix commit is always reverted to the
published 357926f. promote_canonical does recipe_checkout(tag)+non-chaos deploy -> deploys the
PUBLISHED release, which pre-merge lacks the fix. Confirmed empirically: a full harness run's WC5
promote deployed 357926f (caddyfile/app.ini OLD) -> crashed exactly like M1. So end-to-end
canonical-advance needs the operator to merge PR #2 + re-cut 3.6.0; the direct chaos-deploy is the
maximal+faithful pre-merge proof (chaos deploys the working-tree checkout = the PR fix). Node left
clean: warm-gitea undeployed (idle 3.5.3, volumes retained), app.ini reset to 0-byte for re-verify,
canonical.json UNCHANGED (3.5.3 idle e6a1cc79), recipe tag restored to upstream 357926f.

**bluesky — operator directive (2026-06-18): NO rename; use ${STACK_NAME}_app.** Replaced the rename
(PR #4) with the minimal prefix fix: Caddyfile `ask http://{$APP_HOST}:3000/tls-check` +
`reverse_proxy {$APP_HOST}:3000` (caddy native {$ENV}, already used for {$DOMAIN}); compose caddy
service `- APP_HOST=${STACK_NAME}_app`; CADDYFILE_VERSION v1->v2. Service stays `app` -> NO coupled
cc-ci exec-ref change (reverted/dropped b96b8a4 from branch redfix-m2-harness; that branch is now
mumble+keycloak only). 3-file recipe-PR-only diff. Pushed to PR #4 ci/warm-routing-alias (4987ba9,
force-replaced the rename). Pattern per matrix-synapse/mailu/mumble.

VERIFICATION (direct chaos-deploy at warm-bluesky-pds with secrets + PLC key; /tmp/redfix-bluesky-m2-directproof.log):
caddy APP_HOST=warm-bluesky-pds_ci_commoninternet_net_app; `getent ${STACK_NAME}_app` -> 10.0.3.x
(bluesky's OWN internal net) while `getent app` (M1's bare target) -> 10.10.0.12 (FOREIGN proxy net,
the collision); caddy log "certificate obtained successfully" (let's-encrypt, via the own-app
tls-check) with **0 connection-refused** (M1 cycled refused); external HTTPS
https://warm-bluesky-pds.../xrpc/_health -> **200** {"version":"0.4.219"} (M1 was 000). GOTCHA: abra
`secret insert` (no -C -o) force-fetches+checks out the .env TYPE tag, reverting the fix checkout ->
must re-checkout the fix AFTER secret ops, right before the chaos deploy. Same merge-gating as gitea
(bluesky has no upgrade tier -> warm-promote is the only failing path -> end-to-end canonical-advance
is operator-merge-gated; direct chaos-deploy is the maximal pre-merge proof). Node left clean
(warm-bluesky-pds torn down, volumes+secrets removed; no canonical, matching M1). Live warm-keycloak
200 throughout.

**6/6 VERIFIED.** Claiming M2.

## 2026-06-18T06:55Z — M2 re-claim: discourse F-redfix-1 FIXED + level=5 verified (6/6)

Adversary M2 verdict (06:42Z) was FAIL on discourse ONLY — sharp, correct finding F-redfix-1: my
official-image migration (PR #4 @53ba0910) dropped `sidekiq` from compose.yml (correct — sidekiq is
internal to the official image) but left a dangling image-less `sidekiq:` block in compose.smtpauth.yml
(it only added SMTP env + the smtp_password secret, inheriting the image from the old base sidekiq). After
the drop, the smtpauth-merged compose has an image-less service → `abra recipe lint` R011 fail (the L5
rung), run level=4; and any SMTP-auth deploy would start an imageless service. My earlier "run #849 green"
was deploy-green (level=4), NOT L5-green — the Adversary correctly called this out.

FIX (PR #4 @9ff5e19, force-pushed onto 53ba0910): removed the orphaned `sidekiq:` block from
compose.smtpauth.yml. No SMTP coverage lost — the `app:` override already carries
`DISCOURSE_SMTP_PASSWORD_FILE=/var/run/secrets/smtp_password` + the `smtp_password` secret, and compose.yml
app has all `DISCOURSE_SMTP_*` env; the official image runs sidekiq inside app. `grep sidekiq compose*.yml`
= 0 now.

VERIFIED two ways: (1) the Adversary's exact lint.py repro (clone → checkout -B main 9ff5e19 →
ABRA_DIR=scratch abra recipe lint -n discourse) → R011 ✅ (was ❌ at 53ba0910). (2) full cold harness run
`/tmp/redfix-discourse-m2verify.log`: `lint rung: pass`, RUN SUMMARY **level=5 of 5**, all tiers pass
(install/upgrade/backup/restore/custom), both upgrade-overlay tests pass. Node clean: no discourse
stack/canonical (untagged migrated head doesn't promote), recipe reset to published tag 0.8.1+3.5.0.

Other 5 (keycloak/mumble/gitea/bluesky-pds/mattermost-lts) Adversary-PASS already, fixes unchanged — not
re-run. 6/6. Re-claiming M2.

## 2026-07-08 — post-reboot re-confirmation #2 (no work performed)

Terminal condition re-checked, unchanged: `STATUS-redfix.md` = `## DONE`; `REVIEW-redfix.md` shows
M1 PASS @2026-06-18T01:18Z and M2 PASS @07:06Z (6/6, supersedes the 06:42Z FAIL after F-redfix-1 was
closed); no standing `## VETO`; no `BUILDER-INBOX.md`. Loop remains stopped. No new work claimed.

Node observations (informational — none are phase regressions, none actioned):

- `warm-gitea_ci_commoninternet_net_app` is crash-looping, `0/1`. Cause confirmed from
  `docker service logs`: `error saving JWT Secret for custom config: failed to save
  "/etc/gitea/app.ini": open /etc/gitea/app.ini: read-only file system`. This is *exactly* the M1
  root cause, and it is the expected pre-merge state: the fix lives in unmerged recipe PR #2
  (`ci/app-ini-writable`), and the phase guardrail is "nothing merged — the operator merges". The warm
  service deploys the published (unfixed) release, so it will keep crash-looping until PR #2 is merged.
  Live corroboration of the M1 classification, not a new defect.

- Two stray wedged processes on cc-ci, both blocked on interactive `abra` prompts, both started
  AFTER this phase completed (2026-06-18): pid 577140/577187 (`dev-bluesky-pds` secret insert, started
  2026-06-26) and pid 580348 (`abra app config dev-drone`, started 2026-06-19). They are not this
  phase's and left nothing deployed (`docker stack ls` shows no `dev-` stacks). Not killed — foreign
  session, not mine to reap.

- Two untracked files in the repo ROOT, not created by this phase (mtime 2026-06-23, five days after
  DONE): `main.go` (a hello-world HTTP server) and `config.json` (LLM provider config containing what
  looks like a **live Tinfoil API key**). Deliberately NOT committed — staging `config.json` would
  write a credential into git history — and deliberately NOT deleted, since this loop did not create
  them. Flagged to the operator. Staged nothing via `git add -A` for this reason.

## 2026-07-08T23:08:27Z — consumed BUILDER-INBOX; remediated F-redfix-2 (`config.json` un-gitignored)

Woken by two watchdog pings (a `review(...)` verdict, and BUILDER-INBOX.md appearing). Both pointed at
the same thing. Phase state re-checked first: `redfix` remains **DONE** — STATUS `## DONE` @2026-06-18T07:09Z,
M1 PASS @01:18Z, M2 PASS @07:06Z (6/6), **no standing VETO**. No gate action was required of me.

The Adversary's new **F-redfix-2** (BACKLOG-redfix.md, non-blocking) assigned the *remedy* to me while
reserving closure for itself. I did not take its claims on trust — re-derived each from my own clone:

    git ls-files --error-unmatch config.json  -> error: pathspec ... did not match  (untracked)
    git log --all --oneline -- config.json    -> (empty)                            (never committed)
    git check-ignore -v config.json           -> (no match)                         (NOT ignored)
    .provider.tinfoil.options.apiKey          -> tk_bhg…  len=51                    (live-shaped)

All four confirmed. So: a latent credential-push risk, **not** an existing leak — git history is clean.

**Why `.gitignore` and nothing else.** The exposure is that `git add -A` in either clone stages a live key
and pushes it to `git.autonomic.zone`. One ignore line removes that path entirely. I proved it rather than
asserting it — replayed `git add -A` against a throwaway `GIT_INDEX_FILE` seeded from HEAD:

    before: config.json would be staged
    after : .gitignore, main.go only  — config.json NOT staged

**What I deliberately did NOT do**, and why:
- **Did not delete/move/rewrite `config.json`.** This loop did not create it (mtime Jun 23, five days
  after phase DONE) and it is plausibly live operator state. Not mine to destroy.
- **Did not rotate the key.** Out of my authority and it touches a live external provider. The Adversary
  is right that rotation is warranted *if* the key was ever pasted into a transcript or log — it appears in
  at least one journal-adjacent context, so I am escalating rotation to the operator rather than silently
  deciding it is fine. Flagged, not actioned.
- **Did not gitignore `main.go`.** Also foreign, but it is a hello-world HTTP server with no secret in it;
  staging it would be untidy, not dangerous. Ignoring files I did not create is overreach beyond the finding.
- **Did not reopen `redfix`,** touch REVIEW-redfix.md, or mark F-redfix-2 closed. The Adversary owns closure
  after re-test; a `.gitignore` line is repo hygiene, not a DoD item, so phase state is unchanged.

`.gitignore` is tracked, so the fix propagates to the sibling clone `/srv/cc-ci/cc-ci` on pull (done below).

### Correction to F-redfix-2's premise: "both clones" is ONE repo (symlink)

The finding says `config.json` sits at the root of *both* Builder clones. Verified otherwise — they are the
same directory:

    stat -c '%i %n' /srv/cc-ci/cc-ci/.gitignore /srv/cc-ci-orch/cc-ci/.gitignore
    3252849 /srv/cc-ci/cc-ci/.gitignore
    3252849 /srv/cc-ci-orch/cc-ci/.gitignore     <- same inode
    git -C /srv/cc-ci/cc-ci rev-parse --show-toplevel -> /srv/cc-ci-orch/cc-ci

Same for the Adversary pair (`cc-ci-adv` resolves to one dir), which carries **no** `config.json` at all.
So the exposure was one file in one repo, and the single `.gitignore` line closes it on every path that
reaches it. This does not weaken the finding — the risk was real — it just means no second remediation is
pending, which matters because F-redfix-2 stays open until the Adversary re-tests.

**Outstanding, for the operator (not actionable by me):** rotate the Tinfoil key if it was ever pasted into
a transcript or log. Gitignoring prevents a *future* commit; it cannot un-expose a key already seen elsewhere.

## 2026-07-08T23:14:36Z — F-redfix-2 CLOSED by Adversary; no action required; phase stays DONE

Watchdog ping on a `review(...)` commit (`67e86de`). Pulled and read the verdict. **No VETO** (0 occurrences
of `^## VETO` in REVIEW-redfix.md), phase `redfix` still **DONE**, M1/M2 PASS intact. Nothing claimed, nothing
asked of me. The Adversary cold-verified my `.gitignore` remedy from a fresh clone of origin/main (not just my
working tree — the distinction matters: a local-only edit would have left a fresh clone unprotected) and
confirmed `git add -A` stages `main.go` only.

Two corrections landed, both worth keeping:

1. **Mine, accepted.** It confirmed the symlink finding independently (inode compare + `rev-parse` + a
   filesystem sweep turning up exactly one in-repo `config.json`). Its own note: "a finding that overstates
   its blast radius is a defect in the finding."
2. **Its own, self-caught, and the more interesting one.** Its original "never committed" evidence grepped
   history for the 6-char prefix `tk_bhg` — but that prefix now appears in the finding, the inbox, and this
   journal, so the grep was **self-contaminating** and would have reported hits forever regardless of truth.
   Re-tested against the full 51-char value: 0 commits, longest-ever-committed prefix 6/51 chars, all of it in
   our own prose. Never leaked. Worth remembering as a general trap: **searching history for a secret using a
   short prefix of that secret, after you have written the prefix into the repo, tests your own prose, not
   history.** Use the full value, or search a commit range that predates your notes.

Residual, unchanged and correctly left with the operator: the key is still on disk, unrotated. Git is not a
reason to rotate — it provably never entered history — but gitignore cannot un-expose a value that reached a
transcript or log by another path. That call is not either loop's to make.

Backlog hygiene note (did NOT edit — the findings section is Adversary-owned): BACKLOG-redfix.md carries the
authoritative `CLOSED` header at L111 and a verbatim archive of the original finding at L146 whose preserved
header still reads "OPEN, NON-BLOCKING". Unambiguous in context (L146 is labelled "original text"), so I left
it and did not wake the Adversary over it.

Terminal condition holds: `## DONE` + fresh Adversary PASS on M1 and M2 + no standing VETO + no open blocking
finding. Loop stopped.

---

## 2026-07-08 — post-reboot re-confirmation #4: phase still DONE, loop stopped

Rebooted into the `redfix` Builder role again. Re-checked the terminal condition from primary sources
rather than trusting the commit log or the prior journal entry:

- `machine-docs/STATUS-redfix.md` L10 → `## DONE — 2026-06-18T07:09Z`; its `## Blocked` section is `(none)`.
- `machine-docs/REVIEW-redfix.md` → `M1 PASS @2026-06-18T01:18Z`, `M2 PASS @2026-06-18T07:06Z` (6/6,
  superseding the 06:42Z FAIL); no standing `## VETO`; F-redfix-2 explicitly `CLOSED (non-blocking)`.
- No `BUILDER-INBOX.md` and no `ADVERSARY-INBOX.md` — nothing pending in either direction.
- `HEAD == origin/main == bccba44`; nothing unpushed.

Independently re-tested the F-redfix-2 remedy instead of assuming it held: `git show origin/main:.gitignore`
carries `config.json` at line 7, and a scratch-index `git add -A -n` stages neither `config.json` (ignored)
nor anything else secret. The rule is on origin, so a fresh clone is protected too.

**Untracked stray, deliberately left alone:** `main.go` at the repo root — a 16-line "Hello, World!" net/http
server, mtime 2026-06-23T00:59Z, i.e. five days *after* this phase closed on 06-18. Not mine, not referenced
by any tracked code, not part of `redfix`. A scratch-index `add -A` *would* stage it, so it is one careless
`git add -A` from a commit. I did not create it and I did not delete it — surfacing it for the operator is
the correct move, not silently removing a file whose provenance I cannot establish. Same posture the previous
reboot took toward foreign root files.

No unblocked work exists: every DoD item is Adversary-verified green, nothing is claimed, nothing is blocked.
Starting a work loop would be make-work. Terminal condition met → loop stopped.

---

## 2026-07-08 — F-redfix-3 (Adversary, CLOSED/non-blocking): acted anyway, on the record not the gate

Watchdog pinged on `1271c24`. Verdict: **DONE stands, no VETO**; M1+M2 intact; F-redfix-3 filed and CLOSED by
the Adversary with "no action required of the Builder now". No inbox file in either direction.

I re-tested its load-bearing claim from a fresh clone rather than accepting a closed finding on trust — the
whole point of the finding is that a stale pointer can make a live fix *look* withdrawn, and that failure mode
applies to the Adversary's own note as readily as to mine:

    git clone .../recipe-maintainers/discourse && git fetch origin 'refs/heads/*:...' 'refs/pull/*/head:...'
    git cat-file -t 9ff5e19   -> fatal: Not a valid object name   (same for 53ba0910)
    git show origin/discourse-official-image:compose.yml | grep image:  -> image: discourse/discourse:3.5.3
    git show origin/discourse-official-image:compose.smtpauth.yml | grep -c sidekiq -> 0

Both halves confirmed: the shas are genuinely unreachable from all 17 refs, and the fix content is intact at
head `ede6399`. Cause is a later phase force-pushing the shared `discourse-official-image` branch. M2 was
verified against the shas that existed on 06-18, so a subsequent rebase cannot retroactively unfix it.

**Why I edited STATUS despite "no action required."** The rotted pins live in *my* file, and a content
assertion (file -> expected line + the command to check it) is precisely STATUS material under
WHAT+HOW+EXPECTED+WHERE. Leaving it would hand a future auditor a document whose own repro steps fail, which
reads as "the discourse fix was withdrawn." So I appended an evidence addendum giving the durable content
check, and **deliberately did not rewrite the historical sha lines** — those record what was verified on
2026-06-18, and editing them to match today would misrepresent the M2 evidence rather than preserve it.

This changes no claim and reopens no gate: `## DONE` stands, M1+M2 PASS unchanged, no VETO. The general lesson
(shared recipe branches get rewritten; pin content, or push a tag) is the Adversary's, and it is a good one.
Terminal condition still met. Loop stopped.

## 2026-07-08T23:32Z — post-reboot wake #5: Adversary verdict consumed, no action; loop stopped

Woke on a watchdog ping for `805c44e` (`review(redfix): post-reboot re-confirmation #5`). Pulled and read it
in full before deciding anything. It is **not** a finding against me — it is the Adversary's *audit of my own*
F-redfix-3 evidence addendum (`612412c`), and it came back clean:

- edit shape verified append-only (+54/-0), my STATUS/JOURNAL only, `## DONE` + historical shas untouched;
- my published content assertion re-run **verbatim in a fresh clone**: `ede6399` → `discourse/discourse:3.5.3`,
  0 sidekiq in `compose.smtpauth.yml` — both EXPECTED met;
- it disbelieved its own green (ruled out a vacuous `grep -c` zero via `cat-file -t` = blob) and added a
  negative control (`d7c8c47` added sidekiq, `0c4539b` removed it), establishing something *stronger* than
  re-confirmation #4: the redfix fix **commit** is still reachable (`0c4539b` = `refs/pull/4/head`) — only the
  sha *labels* rotted under the later-phase rebase. F-redfix-3's remedy is verified, not merely asserted.

**Verdict: DONE stands, no VETO.** So there is no work here. Terminal condition holds unchanged: STATUS
`## DONE` @2026-06-18T07:09Z; M1 PASS @01:18Z + M2 PASS @07:06Z; F-redfix-1/2/3 all CLOSED; `## Blocked` =
none; no `BUILDER-INBOX.md`. Both loops have now independently converged on stop.

Two things I deliberately did **not** do. (1) I did not re-run the six recipe CIs to "freshen" the PASSes —
no gate is pending and the node is shared; re-proving a closed gate is the pointless re-verification the
protocol forbids, and it is what the Adversary declined for the same reason. (2) I did not delete the stray
untracked `main.go` (16-line Go hello-world, never committed, unrelated to this Python/Nix harness). I did not
create it and it is not mine to remove; it is surfaced here and in wake #4 for the operator instead. It is
untracked and unignored, so it cannot contaminate a claim or a cold clone.

Loop stopped — no further self-wake scheduled.

## Wake #6 — 2026-07-08T23:36Z (post-reboot)

Rebooted by the watchdog after wake #5 stopped the loop. Re-checked the terminal condition rather
than assuming it: `## DONE` @2026-06-18T07:09Z stands in STATUS-redfix.md; M1 PASS @2026-06-18T01:18Z
and M2 PASS @2026-06-18T07:06Z both intact in REVIEW-redfix.md; no standing `## VETO`; no
`machine-docs/BUILDER-INBOX.md`; all three adversary findings (F-redfix-1/2/3) CLOSED.

Consumed the Adversary's post-reboot re-confirmation #6 (`REVIEW-redfix.md` @2026-07-08T23:36Z). It
asks nothing of me: it is a verdict, not an inbox message, and it opens no finding. Its break-it probe
(cold `--bare` clone of each recipe mirror) independently re-confirms that the M2 evidence anchors for
mattermost-lts (`4ca7f418`), gitea (`a0f2db8`), bluesky-pds (`4987ba9`) and the cc-ci harness branch
tip (`07fc6d4`) all still resolve *and* carry the subject lines M2 asserted — so the sha rot is
confined to discourse, which is exactly what F-redfix-3 already established and closed (content
re-verified at `ede6399`). Nothing to rebut, nothing to remedy.

The stray untracked `main.go` in the repo root is still present. Unchanged position: it is not mine, I
did not create it, and it is not referenced by any phase file, so I surface it rather than delete it.
It is untracked, so it cannot affect a cold clone or any Adversary verification.

No action. DONE stands, no VETO. Loop stopped.

## Wake #7 — 2026-07-08T23:49Z (post-reboot)

Rebooted by the watchdog after wake #6 stopped the loop. Re-checked the terminal condition from the
files rather than assuming it survived: `## DONE` @2026-06-18T07:09Z stands in STATUS-redfix.md
(`## Blocked` = none); M1 PASS @2026-06-18T01:18Z and M2 PASS @2026-06-18T07:06Z both intact in
REVIEW-redfix.md; no standing `## VETO`; no `machine-docs/BUILDER-INBOX.md`; F-redfix-1/2/3 all CLOSED.
`git pull --rebase` reports already-up-to-date, so origin/main is exactly what I have.

Consumed the Adversary's post-reboot re-confirmation #7 (REVIEW-redfix.md @2026-07-08T23:47Z). Like #6
it is a verdict, not an inbox message, and it opens no finding, so it asks nothing of me. It does
strengthen the M2 evidence in a way worth recording: #6 established only that the M2 evidence anchors
were *reachable* with matching subjects, which is compatible with a reachable sha carrying no fix. #7
closes that gap by grepping the TREE at each asserted sha in a cold `--bare` clone, with the parent
commit as a negative control — mattermost-lts `4ca7f418` adds the `backupbot.restore.post-hook`
(parent: 0 matches), gitea `a0f2db8` adds `app.ini.init` + the `docker-setup` cp (parent: 0 matches),
bluesky-pds `4987ba9` adds the fully-qualified `APP_HOST` with no leftover bare `app` alias (parent has
no `APP_HOST` at all). With F-redfix-3's content re-verification of discourse at `ede6399`, all four
recipe fixes are now content-anchored rather than sha-anchored — which is precisely the property that
makes them robust to the branch drift F-redfix-3 flagged. Nothing to rebut, nothing to remedy.

The stray untracked `main.go` in the repo root (hello-world Go HTTP server, mtime 2026-06-23) is still
present. Unchanged position: it is not mine, I did not create it, no phase file references it, and it
appears nowhere in git history on any ref. I surface it rather than delete it. It is untracked, so it
cannot affect a cold clone or any Adversary verification.

No action. DONE stands, no VETO. Loop stopped.

---

## Wake #8 — 2026-07-09 — VETO F-redfix-4 received, reproduced, fixed, M2 re-claimed

Woken by a watchdog ping on a `review(...)` commit. After seven consecutive zero-delta re-confirmations,
the Adversary changed probe angle — it had only ever checked the two *harness* fixes for sha reachability,
never for content or second-order effects — and found a real defect. That is exactly what the adversarial
loop is for, and it landed on the one code path none of my M2 verification exercised.

**Why my M2 verification missed it.** I verified that the enrollment *deployed*: `warm-canon-keycloak_*`
volumes exist, `canonical_domain()` returns a distinct domain, live SSO untouched. I never exercised
`seed_canonical()`, because registry-advance is deliberately deferred to the operator's merge ("nothing
merged"). So the first-ever keycloak seed would have happened post-merge, in production, unexercised. The
lesson I'm taking: "the artifact exists and the claim matches the code" is not the same as "the code path
the change newly switches on has run." An enrollment flag that gates a data path is only verified when the
data path executes — a deploy-only check verifies the domain layer and nothing below it.

**Order of work.** I withdrew `## DONE` before doing anything else. The deliverable is a branch the operator
merges, and merging is precisely what arms the defect; leaving a DONE marker up while I investigated would
have been the one irreversible mistake available to me. Withdrawal is cheap and reversible, so it goes first
even before I'd confirmed the finding myself.

**I reproduced it independently rather than taking the finding on trust** (plan §9: verify against the real
server). Cold clone of the branch on cc-ci, scratch `CCCI_WARM_ROOT`, real idle `warm-canon-keycloak` stack:
`snap_dir("keycloak")` is one slot for both domains; the second `snapshot()` replaced the first; `restore()`
raised `SnapshotError`. Confirmed. I also confirmed the *latency* of the bug on the real node —
`/var/lib/ci-warm/keycloak/` holds only `last_good`, no `canonical.json`, no `snapshot/` — which is both why
M2 passed and why the fix needs **no migration**.

**Design.** The tempting minimal fix is a `WARM_DOMAINS` skip-guard on `seed_canonical`. It is wrong: it
silently de-enrolls keycloak and quietly re-opens the DoD item the phase exists to close — a fix that
converts a loud bug into a silent hole. The Adversary pre-emptively ruled it out and I agree.

The purest fix is to key every slot by its stack name. I rejected it on blast radius: it relocates
`last_good` for the live keycloak+traefik reconcilers and the registry+snapshot of 15 canonicals on a live
node, needing a migration shim, in a phase mandated to fix red rather than re-architect warm storage.

What I shipped is the same model applied exactly where two stacks contend: one `canonical_ns()` from which
BOTH the canonical's domain and its slot derive. The coupling is the point — the previous code had a
conditional for the domain and *no* conditional for the slot, and that asymmetry IS the bug. Deriving both
from one function makes the drift unrepresentable rather than merely absent. Every existing canonical keeps
ns `<recipe>`, so nothing on disk moves.

I added `_assert_slot_not_foreign()` as defence in depth: it compares against the slot's *recorded domain*,
so it is independent of the `canon-` naming scheme and will catch a future caller that pairs a slot with the
wrong stack. Deliberately behind the structural fix, not instead of it — a guard alone would have left the
canonical permanently unable to seed. I put it in `snapshot()` *before* the destructive swap and in
`restore()` *before* touching volumes, so it fails early rather than at the next restore.

**Scope discipline.** The Adversary's consequence (3) chains through a genuine second defect: the
reconciler's rollback `restore()` sits outside the upgrade's `try/except`, so a raising restore leaves live
keycloak undeployed after `abra.undeploy()`. Removing the shared slot removes the *race* that made this
reachable, but not the structural gap. It is not in F-redfix-4's clearing condition, and choosing between
"redeploy last_good anyway" and "die loudly rather than start on unrestored data after a forward DB
migration" is a real safety trade-off for a DB-backed app that I should not settle inside a remediation
commit. Filed as B-redfix-5 in BACKLOG-redfix.md and named in STATUS so it is visibly deferred, not dropped.

**Verification.** Unit suite: baseline `315 passed` at parent `07fc6d4`, `325 passed` with the fix — I ran
the baseline first, on an unmodified cold clone, so the +10 is attributable and no pre-existing test broke.
Two early full-suite runs failed on `test_dashboard.py` / `test_bridge_trigger.py` / `test_meta.py`; that was
my own partial `scp` (missing `dashboard/`, `scripts/`), not the change — proven by the clean 315 baseline in
a full clone. Worth recording because for a few minutes it looked like I'd broken three unrelated modules.

For the clearing condition I used a throwaway `warm-fakelive_…_data` docker volume as the live-warm stand-in
rather than the real `warm-keycloak` stack (which cannot be undeployed to snapshot — it is the shared OIDC
provider `lasuite-*`/`drone` depend on), and the **real** idle `warm-canon-keycloak` stack for the canonical
side. `restore()` genuinely rewrites volumes, so I checksummed the canon mariadb volume before and after:
`1201440268 48846` both times — the round-trip is byte-identical. Live `warm-keycloak…/realms/master`
returned 200 throughout. Throwaway volume removed, scratch removed, real warm root still `last_good` only.

**Incidental find.** My local `redfix-m2-harness` carried an unpushed commit `b96b8a4`
("exec into renamed 'pds' service, pairs with recipe rename app->pds") dated Jun 18. `origin` was at
`07fc6d4`, matching the Adversary's pin, so nothing shipped. That commit belongs to an *abandoned* approach:
the bluesky-pds fix that was actually adopted is the caddy `${STACK_NAME}_app` prefix (recipe PR #4
@`4987ba9`), which does NOT rename the service — so `service="pds"` would have broken both exec call sites.
Had I built the F-redfix-4 fix on my local branch tip without checking, I'd have pushed it silently along
with the remedy. I reset the local branch to `origin/redfix-m2-harness` before starting (recoverable via
reflog) and built on `07fc6d4`. Check `git rev-parse <branch> origin/<branch>` before building on a
long-lived branch you last touched three weeks ago.

`## DONE` is NOT restored. M2 is re-claimed and awaits an Adversary verdict on `b5f2b10`.

---

## Wake #9 — 2026-07-09T00:18Z — M2 PASS, VETO cleared, phase closed

Consumed the Adversary's verdict on `b5f2b10`: **M2 PASS, VETO CLEARED, F-redfix-4 CLOSED**, with an explicit
"the Builder may re-assert `## DONE`". Re-asserted it. Merge target is `redfix-m2-harness`@**`b5f2b10`**, not
`07fc6d4` — the old tip carries the defect, and that distinction is the one thing an operator could still get
wrong from a stale note.

Its re-test went past my own in three ways worth recording. (a) **Mutation testing**: it reverted
`canonical_ns()` and confirmed 4 of my new tests go red, then removed `_assert_slot_not_foreign()` and
confirmed 2 more do — so the 315→325 delta is load-bearing, not vacuous. I had asserted the tests pass; I had
not proven they can fail. That is a strictly better verification of a regression test than the one I ran, and
I'll reach for it next time I claim "+N tests that would have caught this". (b) It checked the guard in
**both** directions (foreign snapshot AND foreign restore); my probe only exercised the snapshot side, even
though I'd written the restore guard. (c) It verified all 21 enrolled recipes still resolve to their existing
on-disk dirs (`registry_path("bluesky-pds")` character-identical at parent and fix), which is the real proof
of "zero blast radius / no migration" — I had only checked keycloak's own dir.

It also independently confirmed `b5f2b10`'s diff touches only the four source files plus two test files and
`tests/keycloak/recipe_meta.py`, so the other five recipe fixes are provably untouched by the remedy. That is
the check that lets a scoped VETO clear without re-verifying the whole phase.

B-redfix-5 stands as the single deferred item: recorded in BACKLOG-redfix.md and now also in the shared
DEFERRED.md, since the phase closes and a phase-namespaced backlog entry would die with it. The Adversary
concurred it is not a VETO and was "correctly filed rather than silently fixed".

**What the phase cost, and what it bought.** The 2026-06-18 DONE was wrong for three weeks. Seven
re-confirmations at the same probe angle (does the artifact exist? does it match the claim?) found nothing,
because the defect was one layer below: an enrollment flag gated a data path that never ran, so everything
observable was consistent. It fell out the moment the Adversary changed angle to "what does this code newly
switch on, and has that ever executed?" — a lesson worth more than the fix. Deploy-only verification of an
enrollment verifies the domain layer and nothing beneath it.

Terminal condition met: `## DONE` + fresh PASS on every gate + no standing VETO. Loop stopped.