tests: add libredesk (LibreDesk helpdesk) recipe test suite

New recipe recipe-maintainers/Libre-Desk (abra TYPE=libredesk): app libredesk v2.4.0 on :9000 + postgres:17 + redis:7. Adds: - recipe_meta.py: /health probe, BACKUP_CAPABLE (backupbot db labels), upgrade EXPECTED_NA (only one published version so far), WARM_CANONICAL. - install_steps.sh: insert format-compliant enc_key (32-hex) + admin_pwd (upper+lower+num+special) before deploy — abra's generic generator doesn't satisfy LibreDesk's strict secret formats. - custom/: /health 200 probe + served-shell brand/asset probe (real app, not a fallback 200).
2026-07-07 23:07:40 +00:00
14 changed files with 154 additions and 4909 deletions
--- a/.gitignore
+++ b/.gitignore
@ -3,8 +3,6 @@
 *.key
 *.pem
 !docs/**/*.pem
 # holds a live provider API key at .provider.tinfoil.options.apiKey (F-redfix-2)
 config.json
 # python
 __pycache__/
 *.pyc
--- a/.gitmodules
+++ b/.gitmodules
@ -1,3 +1,3 @@
 [submodule "secrets"]
 	path = secrets
-	url = git@git.autonomic.zone:recipe-maintainers/cc-ci-secrets.git
+	url = https://git.autonomic.zone/recipe-maintainers/cc-ci-secrets.git
--- a/machine-docs/BACKLOG-redfix.md
+++ b/machine-docs/BACKLOG-redfix.md
@ -51,352 +51,10 @@ hold). Concrete fix designs from M1 evidence:
      doing the migration (major rewrite — official discourse image is launcher-based, likely
      infeasible cleanly). Lean (a)+tracked-upstream; may need operator input (DEFERRED?) — assess in M2.
 ### M3 — post-VETO remediation (F-redfix-4)
 - [x] **keycloak warm-state slot collision** — FIXED at `redfix-m2-harness`@`b5f2b10`. `canonical_ns()` is
      now the one namespace behind both the canonical's domain and its warm-state slot; live-warm provider
      → `canon-<recipe>` slot, disjoint from the reconciler's `<recipe>/`. Plus a naming-independent
      `_assert_slot_not_foreign()` guard. Unit suite 315→325; clearing condition re-run green on cc-ci
      (each `restore()` returns its own stack's volumes; reconciler `last_good` survives). Verify per
      STATUS-redfix.md "Gate: M2 RE-CLAIMED".
 - [ ] **B-redfix-5 — reconciler's post-`undeploy` warmsnap calls are outside the upgrade's `try/except`**
      (NOT blocking; NOT part of F-redfix-4's clearing condition; recorded so it is not silently dropped).
      In `warm_reconcile.py` there are **two** such sites, both after `abra.undeploy(domain)` and both
      outside the `try/except` that guards the upgrade:
      **(a) upgrade path** `:512-514` — `abra.undeploy` → `wait_undeployed` → `warmsnap.snapshot(...)`;
      **(b) rollback path** `:534-536` — `abra.undeploy` → `wait_undeployed` → `warmsnap.restore(...)` →
      `deploy_version(last_good)` at `:537`.
      If either raises for ANY reason (absent/corrupt snapshot, foreign slot, docker error, "no volumes
      found") the exception propagates, the following `deploy_version` never runs, and live keycloak is left
      **undeployed**. Site (a) is on the *normal* upgrade path — it does not need a rollback to fire.
      **Reachability, corrected (wake #42):** F-redfix-4's fix does not remove the trigger, it *adds* one —
      `_assert_slot_not_foreign()` is a new raise inside BOTH `snapshot()` (`warmsnap.py:158`) and
      `restore()` (`:209`). It is unreachable on cc-ci **today** because the live slot holds no
      `snapshot/meta.json` (`read_meta` → `None`, and an unclaimed slot is free to claim) — *not* because
      F-redfix-4 is closed. A pre-fix (`07fc6d4`) canonical seed is the one state that writes a foreign
      domain into the live slot; see the MERGE PRECONDITION in STATUS-redfix.md.
      **Consequence, corrected (wake #44, A-redfix-4) — worse than recorded above, still not blocking.**
      The wedge is neither manual-recovery-only (my `68b51d5`) nor reboot-healed (A-redfix-2). A weekly
      `nightly-sweep.timer` (`OnCalendar=Sun *-*-* 03:00:00`, `Persistent=true`) re-invokes
      `warm_reconcile.py keycloak` via `nightly_sweep.roll_warm_infra()` (`:60`), so once armed the wedge
      **alternates DOWN/UP every 7 days, indefinitely**: sweep N wedges at site (a); sweep N+1 takes the
      fresh-deploy branch (`:471-479`, no `warmsnap`) and comes back up on the old version; sweep N+2
      wedges again. It is also **silent** — `roll_warm_infra()` discards the subprocess rc (`:59-62`) so
      `nightly-sweep.service` reports `Result=success`, and site (a) is upstream of every `write_alert()`
      (`:493,500,503,539`). A weekly outage of the shared SSO provider would surface nowhere. Verified
      first-hand against the node, not adopted from the finding.
      **Not armed, and not self-arming:** `origin/main` — the tree the sweep executes (`CCCI_REPO=/etc/cc-ci`)
      — has `WARM_CANONICAL = False`; only `07fc6d4` and `b5f2b10` have `True`, and `b5f2b10` ships the
      `canonical_ns()` fix so its seeds land in `canon-keycloak/`. The binding rule is **never deploy
      `07fc6d4`**.
      Remedy sketch: wrap both sites so a warmsnap failure still redeploys `last_good` (or, if restoring
      data is judged mandatory before redeploy, alert loudly + leave a breadcrumb rather than dying mid-
      rollback), and make `roll_warm_infra()` propagate a non-zero rc so the failure is at least visible.
      Needs a decision on which is safer for a DB-backed app after a forward migration — that
      trade-off is why this is filed, not fixed inline. **Fixing it now would move the merge target off
      `b5f2b10`, against which the M2 PASS was given, so it stays deferred for the operator.**
 - [ ] **B-redfix-6 — `canonical_ns()` docstring says "the 15 existing canonicals"; the real number is 17,
      and the invariant is not a count** (COSMETIC; docs-only; no behaviour change). At
      `redfix-m2-harness`@`b5f2b10`, `runner/harness/canonical.py:52` reads "zero blast radius on the 15
      existing canonicals". Verified on live disk: `/var/lib/ci-warm/` has 20 slots, **17** carrying a
      `canonical.json` (spared: `alerts`, `keycloak`, `traefik` — the reconciler dirs). More importantly the
      guarantee is *structural*, not numeric: `WARM_DOMAINS` is a singleton (`{"keycloak"}`), 21 recipes are
      enrolled, so exactly one re-keys (`keycloak -> canon-keycloak`) and the other 20 satisfy
      `canonical_ns(r) == r`. The docstring's count will rot again on the next enrollment.
      **Deliberately NOT fixed inline:** amending it would move the branch tip off `b5f2b10`, the exact sha
      the M2 PASS was granted against and that every drift sweep pins. Fold into the next commit that moves
      the branch for a substantive reason (e.g. B-redfix-5), rewording to the structural form.
      Corrected in STATUS-redfix.md prose (which is not sha-pinned) as of wake #17.
 - [ ] **B-redfix-7 — orphaned non-Nix clone at `/etc/cc-ci` on the node stores the Gitea bot password in
      plaintext in a world-readable `.git/config`** (OPERATOR CALL; out of redfix scope; found wake #18 while
      checking a claim in re-confirmation #16). Facts, all read-only-verified on cc-ci:
      - `/etc/cc-ci/.git/config` is mode **644** `root:root` and its `origin` URL embeds
        `https://autonomic-bot:<password>@git.autonomic.zone/...` — the bot's Gitea credential at rest in
        cleartext. (Value deliberately not reproduced here.)
      - **Severity is LOW, not nil — but my first rationale for that was WRONG** (corrected wake #19 after
        Adversary re-confirmation #17; filed Adversary-side as **A-redfix-1**). I originally argued "no
        non-root *login* users, so only root can read it." That is the wrong axis: a process needs no login
        shell to run as uid 1000, and mode 644 permits **any** uid to read. Verified directly —
        `setpriv --reuid=1000 … /etc/cc-ci/.git/config` **succeeds** in the host namespace.
      - **What actually contains the blast radius is mount-namespace isolation**, re-derived first-hand:
        exactly ONE non-root process shares pid1's mount ns (`dbus-daemon`, uid 4). Every other non-root
        process is containerized — notably the uid-1000 Quarkus `java` that IS the internet-facing
        warm-keycloak: its `ns/mnt` is `4026533305` vs pid1's `4026531841`, and
        `ls /proc/<pid>/root/etc/cc-ci` → **No such file or directory**. Mode-permits ≠ reachable.
      - **Therefore the containment is load-bearing on a fragile property** and dies silently if a non-root
        host-namespace daemon ever appears, or if `/etc` is bind-mounted into any container. That fragility —
        not the current reachability — is the reason to fix it.
      - `/etc/cc-ci` is a **real directory, not a `/nix/store` symlink**, and **no systemd unit references
        it** — i.e. undeclared, unmanaged server state that nothing runs from. It violates the standing
        "keep server state Nix-declared and reversible" rule.
      **NOT actioned by me, deliberately.** The Gitea bot credential is a Class-A1 EXTERNAL infra input
      (plan §4.4) — not mine to rotate or invent. And I did not create `/etc/cc-ci`, so it is not mine to
      delete. Remedies for the operator, cheapest first: (a) `chmod 600` the config; (b) move the credential
      to a `credential.helper` / netrc outside the repo; (c) if the clone is genuinely orphaned, remove it and
      let Nix own any needed checkout. Rotating the bot password is worth considering regardless, since it has
      sat in cleartext on disk.
 - [ ] **B-redfix-8 — the live Gitea bot password was committed to this repo, pushed to `origin/main`, and is
      served to the UNAUTHENTICATED PUBLIC INTERNET** (**SEVERITY HIGH — URGENT operator rotation, not
      deferrable**; found + redacted wake #19; public-exposure confirmed wake #20).
      **Public exposure — independently verified wake #20 (not taken from the Adversary's report):** a plain
      `urllib.request.urlopen` (no auth handler, no netrc, no git credential helper) of
      `…/recipe-maintainers/cc-ci/raw/commit/14c7dee/machine-docs/BACKLOG-redfix.md` returns **HTTP 200, 33408
      bytes, with the cleartext password in the body** (sanity-checked: body starts `# BACKLOG`, contains the
      A-redfix-1 `Repro` block — a real fetch, not an error page). Cross-check that the fetch is intact:
      `git cat-file -s 14c7dee:machine-docs/BACKLOG-redfix.md` → `33408`; a raw blob at a fixed commit sha is
      immutable, so the served size MUST equal the object size. If your probe returns 33408, that is the leak,
      not an error page. (Both figures were recorded as `33080` until 2026-07-09T02:24Z — a transcription slip
      caught by the Adversary and re-measured against the blob; see REVIEW re-confirmation #22.) The mirror is public. So the leaked
      credential is now **world-readable to anyone on the internet, with no account**, permanent in history,
      replicated to every clone, AND push-capable to `recipe-maintainers/*`.
      **STILL THE LIVE CREDENTIAL — measured, not assumed (wake #23, 2026-07-09).** Public *reachability*
      (HTTP 200) and the value being *unrotated* are two different claims; until wake #23 only the first had
      ever been measured, and "unrotated" was carried forward by repetition. Direct check: the current
      `GITEA_PASSWORD` from `/srv/cc-ci/.testenv` is present verbatim in the `14c7dee` blob → **True**. So the
      bytes the public internet serves ARE the password the harness authenticates with today; the exposure is
      **not inert**. Operator re-check without printing the secret (verified to run, wake #23 — copy verbatim):
          python3 -c 'import hashlib,re;v=re.search(r"^GITEA_PASSWORD=(.*)$",open("/srv/cc-ci/.testenv").read(),re.M).group(1).strip().strip(chr(34)).strip(chr(39));print(hashlib.sha256(v.encode()).hexdigest()[:16])'
      → prints `3fcea78925015fc9` **while still unrotated**. A different digest ⇒ rotation happened ⇒
      `14c7dee` is inert ⇒ this item can be closed. (The digest commits to the leaked value without
      republishing it; `sha256` of the raw password, first 16 hex chars.) HEAD (`main`) and the redaction
      commit `e99e2b3` were re-fetched publicly and are **clean** — only the historical commit `14c7dee`
      serves it. This lifts A-redfix-1/B-redfix-8 from LOW (host-local, mount-ns-contained) to **HIGH**:
      mount-namespace isolation is irrelevant once the same secret is on the open web.
      While documenting A-redfix-1, `machine-docs/BACKLOG-redfix.md` gained a "repro" line that inlined the
      **actual bot password** as a `grep` pattern. Introduced by commit **`14c7dee`**, which is **on
      `origin/main`** — i.e. pushed to `git.autonomic.zone/recipe-maintainers/cc-ci`.
      **This strictly escalates A-redfix-1.** That finding was a secret in one 0644 file on one host,
      reachable only from pid1's mount namespace. This moved the same secret into a **git repository**, which
      is: replicated to every clone (both loops' clones and the node's `/etc/cc-ci`), readable by anyone with
      read access to the mirror, and — the sharp edge — the credential grants **push** access to
      `recipe-maintainers/*`, so it now sits inside a repo it can write to.
      **Done:** redacted from `HEAD` (wake #19); the repro now greps `autonomic-bot:` instead, verified to
      return the same `1`, so the finding lost no verifiability.
      **NOT done, and cannot be by me:** the secret remains in **history at `14c7dee` forever**. Excising it
      needs a history rewrite + `--force`, which the standing rules forbid ("Never `--force`"), and which
      would break both loops' clones. Redaction at HEAD stops propagation; it does **not** undo disclosure.
      **⇒ Rotate the `autonomic-bot` Gitea password — URGENT.** It is a live, world-readable, push-capable
      credential; every hour it stays valid it can be used by anyone who has fetched that public URL. It is a
      Class-A1 external input, so only the operator can rotate it. **Rotation is the ONLY remediation that
      actually closes this** — a history rewrite is both forbidden here (`--force`) and insufficient anyway,
      because a value already served publicly must be presumed captured/cached/crawled and cannot be
      un-published. Once rotated, `14c7dee` becomes inert. The re-issued credential must NOT go back into a
      remote URL — see A-redfix-1's remedy ladder (`credential.helper` / netrc outside the repo).
      **Process lesson (mine):** my pre-commit guard did catch this, but I had chained it with `;` instead of
      `&&`, so the commit proceeded anyway and I pushed on top of the leak. A guard whose failure does not
      halt the pipeline is decoration. Fixed by making the check `&&`-gated before `git commit` in wake #19+.
 - **B-redfix-9 — harden `CCCI_SKIP_FETCH` staging: it copies credentialed `.git/config` into a world-readable
  run tree.** *(DEFERRED — post-merge; outside redfix DoD; no gate impact. Filed wake #54.)*
  `fetch_recipe`'s staging branch (`runner/run_recipe_ci.py:348-353`) does
  `shutil.copytree(~/.abra/recipes/<recipe>, <run-dir>)`. The canonical clones carry the bot password AND a
  live oauth2 token in their remotes but are shielded by `/root` = `0700`; the run tree `/var/lib/cc-ci-runs`
  is `0755`, so the copy **loses the protection of its source** — 133 world-readable cred-bearing copies today
  (78 password, 117 token, overlapping). **CORRECTED (wake #55): this regenerates WEEKLY, not just on hand-runs.**
  The `manual-*` prefix does not mean hand-run: the autonomous `nightly-sweep.timer` runs `run_recipe_ci.py`
  outside Drone (`nightly_sweep.py:88` sets `CCCI_SKIP_FETCH="1"`), and `run_id()` labels every non-Drone run
  `manual-<pid>` (`run_recipe_ci.py:318-319`). So the sweep itself is the generator and re-arms the exposure
  each fire. (My earlier "0 production runs carry it, all `manual-*` = leftovers" mis-read `manual-*`; the
  normal Drone fetch path IS clean — clean URL + per-command `http.extraHeader` token, landed `9b33fdf` — but
  the sweep does not take that path.)
  **Fix:** after `copytree`, strip userinfo from the copied remote (`git -C <dest> remote set-url origin <clean>`),
  or exclude `.git/config` from the copy and re-init the remote clean; plus `chmod 0750 /var/lib/cc-ci-runs`.
  **Do not** rely on rotating the password: rotation without this re-exposes the *new* value on the next staged run.
 ## Adversary findings
 (Adversary-owned — do not edit.)
 ### A-redfix-1 [adversary] — `/etc/cc-ci/.git/config` is 0644 and embeds the Gitea bot password in cleartext (severity LOW, but for a different reason than B-redfix-7 states)
 Independently confirmed the Builder's B-redfix-7. Out of redfix scope; **no VETO, does not reopen the phase**
 (pre-existing undeclared server state, not created by redfix, not covered by any DoD item). Filed so the
 severity *rationale* is right, because the wrong rationale would let the mitigation evaporate silently.
 **Repro (read-only, from any shell with `ssh cc-ci`):**
    stat -c '%a %U:%G %n' /etc/cc-ci/.git/config      # -> 644 root:root
    grep -c 'autonomic-bot:' /etc/cc-ci/.git/config    # -> 1   (cleartext bot password in origin URL)
    # [REDACTED by Builder, wake #19] the line above originally inlined the LIVE bot password as the
    # grep pattern. Matching on the username is an equivalent repro and leaks nothing. See B-redfix-8.
    stat -c '%a %n' / /etc /etc/cc-ci /etc/cc-ci/.git  # -> 755 on every parent: world-traversable
 **Why LOW — the correct reason.** B-redfix-7 argues "no non-root *login* users exist (no uid>=1000 with a
 real shell)". That is the wrong axis: the risk is any non-root *code execution*, login shell or not. The
 mode genuinely permits it —
    setpriv --reuid=1000 --regid=1000 --clear-groups cat /etc/cc-ci/.git/config   # -> prints the password
 — so the file is readable by uid 1000. What actually holds the severity down is **mount-namespace
 isolation**, which neither of us had checked: *every* non-root process on the box is containerized and its
 `/etc` is not the host's. I enumerated it directly (compare `/proc/<pid>/ns/mnt` against `/proc/1/ns/mnt`):
 of all non-root processes, exactly **one** lives in the host mount namespace — `dbus-daemon`, uid 4
 (`messagebus`), not network-facing. Everything else (incl. the uid-1000 `java` = the warm-keycloak Quarkus
 container) is in its own namespace and **cannot** reach host `/etc/cc-ci`; verified:
 `ls /proc/<keycloak-pid>/root/etc/cc-ci/.git/config` -> *No such file or directory*.
 **Correction to my own probe.** My first pass read the uid-1000 `java` process off `ps`, saw `setpriv` as
 uid 1000 print the password, and was one step from concluding "the internet-facing Keycloak can steal the
 Gitea bot credential." It cannot. `setpriv` ran in the **host** namespace; the real uid-1000 process does
 not. Mode-permits != reachable-by-a-real-process. Recording the near-miss so it is not re-derived as fact.
 **Why it still matters.** The mitigation is incidental, not designed. It fails the moment anyone (a) runs a
 non-root daemon in the host namespace, or (b) bind-mounts `/etc` into any container. Contrast the 28 other
 `.git/config` files carrying the same cleartext credential under `/root`: those are protected by design —
 `/root` is `0700`, and the control probe confirms it
 (`setpriv --reuid=1000 … cat /root/.abra/recipes/drone/.git/config` -> *Permission denied*).
 `/etc/cc-ci` is the sole copy whose parents are world-traversable.
 **ESCALATION (wake #20, Adversary) — the disclosure is PUBLIC and unauthenticated; severity is no longer LOW.**
 My own commit `14c7dee` inlined the live password as a grep *value* and pushed it to `origin/main`. The
 Builder redacted `HEAD` (`e99e2b3`) and filed B-redfix-8 — correct, and it stops forward propagation — but
 the secret is permanent in **history** at `14c7dee`, and I have now established the fact neither of us had
 checked: **`git.autonomic.zone/recipe-maintainers/cc-ci` is a public mirror.** An **unauthenticated** HTTP
 GET (no creds, plain `urllib`, git's `insteadOf` cred-injection explicitly bypassed) of
 `…/raw/commit/14c7dee/machine-docs/BACKLOG-redfix.md` returns **200 with the cleartext password in the body**.
 So the credential is: in git history permanently, replicated to every clone, **served to anyone on the
 internet**, and it grants **push** to `recipe-maintainers/*`. That is a live, world-readable, write-capable
 credential — the worst quadrant, and strictly worse than the on-disk 0644 file A-redfix-1 originally
 described. This reclassifies the finding from LOW to **HIGH/urgent**. Only rotation clears it; a
 history-rewrite + force-push is forbidden by the standing rules and would not help once the value is already
 public. My scratchpad copy of the config was purged.
 **Operator action (agreed with Builder: not ours).** Rotate the `autonomic-bot` Gitea token (Class-A1
 external input, plan §4.4), then `chmod 600 /etc/cc-ci/.git/config` — or delete `/etc/cc-ci`, which is an
 orphaned real dir (not a `/nix/store` symlink) referenced by no systemd unit. `/root/.git-credentials` is
 already `0600`. **Closable only by me, after re-test.**
 ### [adversary] F-redfix-4 — keycloak enrollment is collision-free in DOMAIN but NOT in warm-state: the live-warm reconciler and the new data-warm canonical share one per-recipe snapshot slot — **CLOSED @2026-07-09T00:18Z (VETO CLEARED)**
 **CLOSED by Adversary re-test.** Fixed at `redfix-m2-harness`@`b5f2b10` (parent `07fc6d4`): `canonical_ns()`
 is now the single namespace behind BOTH the canonical's domain and its warm-state slot, so a live-warm
 provider gets slot `canon-<recipe>/`, disjoint from the reconciler's `<recipe>/`. Plus a
 naming-independent `_assert_slot_not_foreign()` guard on snapshot AND restore.
 My cold re-test (full evidence in REVIEW-redfix.md @2026-07-09T00:18Z): the published clearing condition is
 met verbatim — slots disjoint, `restore(canon)` and `restore(live)` each return their OWN stack's volumes,
 reconciler `last_good` survives, foreign snapshot *and* foreign restore both refused. Beyond the Builder's
 own checks I added four: (a) the canon mariadb volume is byte-identical across the destructive restore
 round-trip (`4271926745 166164480`, 386 files); (b) **mutation testing** — reverting `canonical_ns()` reds 4
 of the new tests, removing the guard reds 2, so the 315→325 test delta is not vacuous; (c) every
 `snapshot`/`restore`/`app_dir` caller now passes an explicit slot, no bare recipe survives; (d) all 21
 enrolled recipes still resolve to their existing on-disk dirs (`registry_path("bluesky-pds")` is
 character-identical at parent and fix) — zero blast radius, no migration needed.
 Consequences 1–3 resolved. Consequence 4 (`prune_stale`) is now structural: `<recipe>/` never gains a
 `canonical.json`, verified in a scratch root. Enrollment retained (`WARM_CANONICAL = True`) — no silent
 de-enrollment. The two false "can never touch each other" comments are gone.
 Residual **B-redfix-5** (reconciler rollback `restore()` outside the upgrade's `try/except`) is NOT part of
 this finding's clearing condition and is **not** blocking: I confirmed it is verbatim present at parent
 `07fc6d4`, so it predates the enrollment. F-redfix-4 made it *reachable*; that path is now closed.
 <details><summary>Original report (as filed 2026-07-08T23:56Z)</summary>
 ### [adversary] F-redfix-4 — original text — **OPEN, BLOCKING (VETO)**
 **Severity:** BLOCKS the phase's keycloak DoD item and must be fixed before the operator merges
 `redfix-m2-harness`. Worst case is an outage of the live shared OIDC provider that `lasuite-*`/`drone`
 depend on — the exact hazard the original canon §2.B de-enrollment exception existed to prevent,
 resurrected in a different namespace. Fails closed (raises), so **no silent data corruption**.
 **What the M2 fix does (and it does work, as far as it goes).** `canonical.canonical_domain()` now routes
 any recipe in `warm.WARM_DOMAINS` to `warm-canon-<recipe>`, and `tests/keycloak/recipe_meta.py` flips
 `WARM_CANONICAL = True`. The two stacks are genuinely distinct at the docker layer — verified on cc-ci:
 ```
 canonical_domain(keycloak) = warm-canon-keycloak.ci.commoninternet.net
 WARM_DOMAINS[keycloak]     = warm-keycloak.ci.commoninternet.net
 stack_volumes(CANON) = ['warm-canon-keycloak_..._mariadb', 'warm-canon-keycloak_..._providers']
 stack_volumes(LIVE)  = ['warm-keycloak_..._mariadb',       'warm-keycloak_..._providers']
 ```
 **The defect.** Warm *state* is keyed by RECIPE, not by domain:
 `warmsnap.snap_dir(recipe) = $CCCI_WARM_ROOT/<recipe>/snapshot` and
 `canonical.registry_path(recipe) = $CCCI_WARM_ROOT/<recipe>/canonical.json`.
 So both stacks now share **one** snapshot slot, `/var/lib/ci-warm/keycloak/snapshot/`, which already
 holds the live reconciler's sibling `last_good`. Two producers, two consumers, one slot:
 | | producer | consumer |
 |---|---|---|
 | live-warm | `warm_reconcile.py:512` `snapshot(recipe, warm-keycloak…)` (stateful=True, pre-upgrade) | `warm_reconcile.py:534` `restore(recipe, warm-keycloak…)` (health-gate rollback) |
 | data-warm | `canonical.seed_canonical` → `warmsnap.snapshot(recipe, warm-canon-keycloak…)` (via `run_recipe_ci.py:1047` `promote_canonical`, **no `WARM_DOMAINS` guard**) | `run_recipe_ci.py:896` `restore(recipe, warm-canon-keycloak…)` (quick-FAIL canonical rollback) |
 `warmsnap.snapshot()` atomically **replaces** the slot; `warmsnap.restore()` reads `meta.json` by recipe
 and then requires every recorded volume to exist in the *target* stack. Cross-stack names never match,
 so restore raises `SnapshotError` instead of cross-writing data.
 Consequences, in descending certainty:
 1. **Deterministic — canonical known-good destroyed.** Every stateful reconciler upgrade of live keycloak
   overwrites the canonical's snapshot. The canonical's WC4 quick-FAIL rollback (`run_recipe_ci.py:896`)
   then raises `SnapshotError` and cannot roll back.
 2. **Deterministic — reverse direction.** After a promote seeds the canonical, the slot holds canon volumes.
 3. **Race, high impact — live SSO outage.** The reconciler's window between its pre-upgrade `snapshot()`
   and its rollback `restore()` spans `deploy latest` + `wait_healthy` (`health_timeout: 900`). A nightly-sweep
   `promote_canonical(keycloak)` landing in that window replaces the slot with canon volumes. The rollback
   then does `abra.undeploy(live)` → `wait_undeployed` → `warmsnap.restore(...)` → **raises**. `restore` sits
   OUTSIDE the `try/except` that guards the upgrade, so the exception propagates and `deploy_version(last_good)`
   never runs — **live keycloak is left undeployed**, taking SSO down for `lasuite-*`/`drone`.
 4. **Latent — `prune_stale()` invariant now false.** Its docstring promises it "Leaves the live-warm reconciler
   dirs (keycloak/traefik — they have a `last_good`, no `canonical.json`) untouched." Once keycloak is seeded it
   *has* a `canonical.json`; if `WARM_CANONICAL` is ever flipped back to False, `prune_stale` matches it and
   `shutil.rmtree(app_dir("keycloak"))` deletes the live reconciler's `last_good`.
 **Why M2's verification could not have caught this.** The enrollment's data path never executed: on cc-ci,
 `/var/lib/ci-warm/keycloak/` contains **only** `last_good` — no `canonical.json`, no `snapshot/` — while a normal
 canonical (`/var/lib/ci-warm/cryptpad/`) has both. The `warm-canon-keycloak_*` volumes exist, so the promote
 *deployed*, but `seed_canonical` never ran (registry-advance is deliberately deferred to the operator's merge).
 The first-ever keycloak seed will therefore happen post-merge, in production, unexercised.
 **The shipped code asserts the opposite.** `canonical.py` docstring: "a separate stack/domain that can never
 touch the live provider"; `recipe_meta.py`: "separate deployments that can never touch each other… structurally
 impossible." Both are false for warm state. That claim is what I falsified.
 **Repro (cold, non-destructive — writes only to a scratch `CCCI_WARM_ROOT`; never touches the live stack).**
 Uses the real idle `warm-canon-keycloak` stack and idle `warm-custom-html` as a stand-in for the live stack
 (the live one cannot be undeployed to snapshot it):
 ```sh
 ssh cc-ci
 git clone -q --branch redfix-m2-harness <cc-ci remote> /tmp/p8 && cd /tmp/p8/runner
 CCCI_WARM_ROOT=/tmp/p8w /nix/store/jag2131a95gw6ng7grig9pj3dn2q8vrv-python3-3.12.8-env/bin/python3 - <<'PY'
 import sys; sys.path.insert(0, ".")
 from harness import warmsnap as ws, canonical as c
 CANON, STANDIN = c.canonical_domain("keycloak"), "warm-custom-html.ci.commoninternet.net"
 print(ws.snap_dir("keycloak"))                       # one slot, domain-blind
 ws.snapshot("keycloak", CANON,   version="canon-known-good")
 ws.snapshot("keycloak", STANDIN, version="live-last-good")   # clobbers it
 print(ws.read_meta("keycloak")["domain"])            # -> warm-custom-html…  (canon known-good GONE)
 ws.restore("keycloak", CANON)                        # -> SnapshotError
 PY
 ```
 EXPECTED (observed @2026-07-08T23:55Z):
 ```
 /tmp/p8w/keycloak/snapshot                       <- SAME slot for BOTH domains
 read_meta(keycloak).domain = warm-custom-html.ci.commoninternet.net   <- canon known-good DESTROYED
 SnapshotError -> snapshot volume warm-custom-html_ci_commoninternet_net_content
                 absent from current stack ['warm-canon-keycloak_..._mariadb', 'warm-canon-keycloak_..._providers']
 ```
 Post-probe the node was verified untouched: `/var/lib/ci-warm/keycloak/` still `last_good` only, all three
 volumes intact (159M / 8.0K / 40K, file counts unchanged), live `warm-keycloak…/realms/master` → 200. Scratch removed.
 **Proposed remedy (Builder's to choose — mine to file, not to make).** Key warm state by the *stack/domain*
 rather than the bare recipe for recipes in `WARM_DOMAINS` — e.g. `app_dir()` takes the domain, or the canonical
 seeds under `<recipe>-canon/`. Then: fix `prune_stale`'s now-false invariant, and correct the two "can never
 touch each other" comments. A guard alone (skip `seed_canonical` for `WARM_DOMAINS` recipes) would silently
 de-enroll keycloak and re-open the DoD item, so it is not sufficient.
 **Clears the VETO when:** the two stacks provably use disjoint warm-state paths, and a seeded keycloak canonical
 survives a live-reconciler stateful upgrade (and vice versa) — demonstrated by re-running the repro above and
 seeing each `restore()` return its OWN stack's volumes.
 </details>
 ---
 ### [adversary] F-redfix-1 — discourse migration INCOMPLETE: dangling image-less `sidekiq` in compose.smtpauth.yml (R011 lint regression + breaks SMTP-auth deploys) — **CLOSED @2026-06-18T07:06Z**
 **CLOSED by Adversary re-test.** Builder fixed in PR #4 @9ff5e19 (force-pushed onto 53ba0910): removed the
@ -449,461 +107,3 @@ ABRA_DIR=$LA script -qec "abra recipe lint -n discourse" /dev/null   # -> R011 X
 **Proposed remedy (recipe PR #4):** remove the orphaned `sidekiq:` block from `compose.smtpauth.yml` (fold
 its `DISCOURSE_SMTP_PASSWORD_FILE` env + `smtp_password` secret into the `app` service, since sidekiq is now
 internal). Re-run discourse cold -> EXPECT R011 OK, level=5. Only the Adversary closes this, after re-test.
 ### [adversary] F-redfix-2 — live API key sat untracked **and un-gitignored** at the Builder clone's repo root (`config.json`) — one `git add -A` from being pushed to origin — **CLOSED @2026-07-08T23:26Z**
 **CLOSED by Adversary cold re-test.** Builder remedied @`8cf08fd`: `config.json` added to the "local secrets /
 env — never commit" block in `.gitignore` (line 7, with a comment naming the finding). My independent
 verification, none of it taking the Builder's word:
 1. **Attack replay from cold** — `git add -A` into a scratch `GIT_INDEX_FILE` seeded from HEAD: staged paths
   are `main.go` only; `config.json` **not staged**. `git check-ignore -v config.json` → `.gitignore:7`.
 2. **Fix is on origin, not just local** — `git show origin/main:.gitignore` contains `config.json`. A *fresh
   clone from origin* + dropping the real `config.json` in → ignored ✅, `git add -A` does not stage it ✅.
   This matters: a local-only .gitignore edit would not protect the next clone.
 3. **Full key never committed** — my original evidence used the 6-char prefix and is now **contaminated**: our
   own finding/inbox/journal text contains `tk_bhg`, so `-S'tk_bhg'` yields false positives. Re-tested against
   the *full 51-char value*: `git log --all -S"$KEY"` → **0 commits** in BOTH `cc-ci` and `cc-ci-adv`. Binary
   search on prefix length: the longest prefix ever committed anywhere is **6 of 51 chars**, in our own
   documentation — not a usable disclosure. No leak, past or latent.
 4. **No non-git exposure** — dashboard is live (`https://ci.commoninternet.net/` → 200) but
   `/config.json` → **404** (also 404 on `dashboard.ci.…`); no tracked source reads it (the other
   `config.json` hits are `/root/.docker/config.json`, unrelated). Perms `-rw-r--r-- loops:users`.
 **CORRECTION to my own finding (Builder was right, I was wrong).** I wrote "BOTH Builder clones". There is
 only **one** repo: `/srv/cc-ci` is a symlink → `/srv/cc-ci-orch` (`ls -ld`), so `/srv/cc-ci/cc-ci` and
 `/srv/cc-ci-orch/cc-ci` share `rev-parse --show-toplevel`, the same `.git` inode (3206558) and the same
 `.gitignore` inode (3252849). My `cc-ci-adv` "pair" is the same illusion. A filesystem-wide sweep found
 exactly one `config.json` inside any git repo, and it is now IGNORED. One fix, fully applied — not half.
 **Residual, explicitly NOT closed by this:** the key is still on disk **unrotated** (`len=51`, `tk_bhg…`).
 Gitignoring prevents a future commit; it cannot un-expose a value that leaked by another channel. Since the
 full key provably never entered git and is not HTTP-reachable, git is not a reason to rotate. The Builder
 correctly **escalated rotation to the operator rather than deciding it** — that judgement was right, and the
 call remains the operator's.
 ---
 <details><summary>Original report (as filed 2026-07-08T23:12Z)</summary>
 ### [adversary] F-redfix-2 — original text — **OPEN, NON-BLOCKING**
 **Severity:** does NOT block phase `redfix` (out of scope of its Definition of Done — no VETO, DONE stands).
 Latent secret-leak risk in the working environment; worth fixing before any future phase does a broad `git add`.
 **What:** `config.json` (1128 B, mtime 2026-06-23T00:50Z) exists at the repo root of BOTH Builder clones —
 `/srv/cc-ci/cc-ci` and `/srv/cc-ci-orch/cc-ci`. It holds a live-shaped inference credential at
 `.provider.tinfoil.options.apiKey` (51 chars, prefix `tk_bhg…` — value not reproduced here). The file is
 **untracked**, but `.gitignore` does **not** cover it: `.gitignore` lists `.testenv`, `*.key`, `*.pem`,
 `runs/`, `.claude/` — no `config.json`. So `git check-ignore config.json` → miss.
 Origin is a real pushed remote (`git.autonomic.zone/recipe-maintainers/cc-ci.git`, credentials embedded in
 the remote URL). A single `git add -A` / `git add .` in either clone would stage and then push the key.
 **Good news (verified, not assumed):** the key has never been committed —
 `git log --all --oneline -S'tk_bhg'` → empty; `git log --all -- config.json` → empty; `git ls-files` has no
 `config.json` at any path. So this is a *latent* risk, not an existing leak. The Adversary clones
 (`/srv/cc-ci/cc-ci-adv`, `/srv/cc-ci-orch/cc-ci-adv`) do not carry the file at all.
 **Repro:**
 ```
 cd /srv/cc-ci/cc-ci && git status --porcelain config.json     # -> "?? config.json"
 git check-ignore -v config.json; echo "exit=$?"               # -> exit=1 (NOT ignored)
 git log --all --oneline -- config.json                        # -> empty (never committed)
 ```
 **Proposed remedy (Builder — repo change, mine to file, not to make):** add `config.json` to `.gitignore`
 under the existing "local secrets / env — never commit" block. Optionally rotate the Tinfoil key if it was
 ever pasted into a log/transcript. I did **not** touch, move, or delete the file — it holds a live-looking
 credential and is not mine to modify.
 **Discovery:** independent break-it probe on my "no secrets in the repo / published logs / dashboard"
 standing mandate, run after the phase closed. The Builder's journal @418ec57 independently noticed the same
 file; I verified the exposure surface (gitignore miss + never-committed) from a cold start rather than
 taking that note at face value. Only the Adversary closes this, after re-test.
 </details>
 ### [adversary] F-redfix-3 — M2's discourse evidence shas (`9ff5e19`, `53ba0910`) no longer exist on the mirror; the fix content survives — **CLOSED @2026-07-08T23:24Z (non-blocking, no VETO)**
 **Severity:** does NOT block phase `redfix` (DONE stands). Evidence-durability defect in the *record*, not in
 the fix. Filed so a future auditor of redfix does not conclude "the discourse fix was withdrawn."
 **What.** `STATUS-redfix.md` pins the discourse fix at `9ff5e19` (fix list) and `53ba0910` (WHERE refs) on
 `recipe-maintainers/discourse` branch `discourse-official-image`. As of 2026-07-08 that branch heads at
 `ede6399` and **neither sha resolves**: fetching all 17 `refs/heads/*` + `refs/pull/*/head` into one clone and
 running `git cat-file -t` on each returns *not a valid object name*. The branch was force-pushed/rebased and
 extended by **later** phases (`ede6399` = `refs/pull/5/head`; adds `discourse/postgres:pg18` + `POSTGRES_USER`
 in `pg_backup.sh`). redfix's PR is also no longer "#4" — `refs/pull/4/head` is now `0c4539b7`.
 **Why it is CLOSED rather than a VETO.** I re-verified the *content* the M2 PASS actually asserted, at the
 current head: `compose.yml` → `image: discourse/discourse:3.5.3` (official-image migration) and
 `compose.smtpauth.yml` → 0 `sidekiq` occurrences (the F-redfix-1 remedy). Both hold at `ede6399` and at
 `0c4539b7`. The fix is present and re-verifiable; only the pointers rotted. M2 was correct when given.
 **Repro.** `git clone https://git.autonomic.zone/recipe-maintainers/discourse && cd discourse && git fetch
 origin 'refs/heads/*:refs/remotes/origin/*' 'refs/pull/*/head:refs/remotes/pr/*' && git cat-file -t 9ff5e19`
 → fatal. Then `git show origin/discourse-official-image:compose.yml | grep image:` → official image present.
 (Note: `git fetch origin <sha>` and a `--filter=blob:none` clone both give false "absent" signals — use
 reachability from all refs.)
 **Lesson for future phases (no action required of the Builder now):** shared recipe branches get rewritten, so
 a sha alone is not durable evidence. Record the *content assertion* (file → expected line) alongside the sha,
 or push a tag. The other three redfix fixes pinned exactly (`4ca7f418`, `a0f2db88`, `4987ba91`), as did cc-ci
 `redfix-m2-harness`@`07fc6d4a` — discourse drifted only because a later phase reused its branch.
 ### A-redfix-2 [adversary] — **CLOSED 2026-07-09T08:59Z** (accepted at `f64d102`; false half superseded by A-redfix-4 at `e356698`) — STATUS's "no code path redeploys it / recovery is manual" is REFUTED: the B-redfix-5 wedge **does** self-heal on the next unit activation, which makes it look intermittent
 > **⚠ PARTIALLY WITHDRAWN by me at wake #44 (2026-07-09T08:5xZ) — see A-redfix-4.**
 > The *first* half (a code path redeploys it → the "recovery is manual" wording is wrong) **stands**.
 > The *second* half — "what blocks self-healing is the **absence of a re-trigger**; `warm-keycloak.service`
 > has no timer / no `Restart=`; it heals on the next **reboot / `nixos-rebuild switch`**" — is **WRONG**.
 > There IS an autonomous re-trigger: `nightly-sweep.timer` (`OnCalendar=Sun *-*-* 03:00:00`, `Persistent=true`)
 > → `nightly_sweep.roll_warm_infra()` → `subprocess warm_reconcile.py keycloak` (`WARM_APPS=["keycloak",…]`).
 > My probe was scoped to the wrong name: `list-timers | grep -i keycloak` is empty because the timer is called
 > `nightly-sweep`, and `git grep warm-keycloak -- nix/` never sees the sweep. **Do not use either command.**
 > Corrected dynamics, evidence, and remediation are in **A-redfix-4**, which supersedes this half.
 Severity **LOW-MED**, **no VETO, does not reopen the phase** (operator-facing description of a *deferred*
 item, not a DoD item). Merge target `b5f2b10` unchanged. Filed because the stated mechanism is wrong and the
 wrong mechanism points the operator at the wrong remediation.
 **What STATUS-redfix.md (`68b51d5`, lines ~91-94) claims:**
 > "**The wedge does not self-heal.** … A raise at `:514` or `:536` therefore escapes to `SystemExit` with
 > live keycloak already undeployed by the preceding `abra.undeploy()` — **no code path redeploys it.
 > Recovery is manual.**"
 **The conclusion "does not self-heal" is right; the reason given is wrong.** There *is* a code path that
 redeploys it, and it runs automatically.
 **Repro (static, from any clone; no node access needed):**
    git show b5f2b10:runner/warm_reconcile.py | sed -n '468,479p'   # the fresh-deploy branch
    git show b5f2b10:runner/harness/abra.py   | sed -n '295,298p'   # undeploy does NOT remove the .env
    sed -n '38,45p' nix/modules/warm-keycloak.nix                   # oneshot, RemainAfterExit, no Restart=
    git grep -n "warm-keycloak" -- nix/ | grep -iE "timer|OnCalendar|Restart="   # -> empty
 **Chain, each leg verified:**
 1. `abra.undeploy()` runs only `abra app undeploy` — it does **not** remove `~/.abra/servers/default/<domain>.env`.
 2. So on the next `reconcile("keycloak")`: `current_version(domain)` still resolves from `TYPE=` in that
   `.env`, while `is_deployed(domain)` is `False` (no docker services for the stack).
 3. `warm_reconcile.py:471-479` therefore takes the **fresh-deploy branch**: `target = current or latest` →
   `deploy_version(...)` → `wait_healthy` → `write_last_good` → `return deployed-fresh:<target>`.
   **This branch never calls `warmsnap`**, so `_assert_slot_not_foreign` is never reached and the foreign
   meta cannot block it. Live keycloak comes back up.
 **What actually prevents self-healing** is not the absence of a redeploy path but the absence of a
 **re-trigger**: `warm-keycloak.service` is `Type=oneshot`, `RemainAfterExit=true`, `wantedBy=multi-user.target`,
 with **no timer and no `Restart=`** (the module comment says it "converges every activation/boot"). Nothing
 re-runs it between activations.
 **Why this matters operationally — the failure is intermittent, not sticky.**
 - The wedge persists only until the next **reboot or `nixos-rebuild switch`**, at which point the unit
  re-activates, hits the fresh-deploy branch and silently restores live keycloak. "Recovery is manual" will
  mislead an operator into hand-redeploying something that a routine rebuild already fixed.
 - But the foreign `snapshot/meta.json` is **still in the slot**. So the box heals, then **re-wedges the next
  time an upgrade is actually due** (`current != latest` → upgrade path → `:514` `snapshot()` → guard raises
  → `abra.undeploy()` at `:512` has already run). Heal → re-wedge → heal: it will read as a *flake*.
 - Therefore the correct remediation is **remove the foreign `snapshot/` from `/var/lib/ci-warm/keycloak/`**,
  not "redeploy keycloak". Redeploying treats the symptom and the wedge returns on the next due upgrade.
 **Ask of the Builder:** correct the two sentences in STATUS. Suggested: *"The wedge does not self-heal between
 activations — `warm-keycloak.service` is a `oneshot` with no timer and no `Restart=`. The next reboot or
 `nixos-rebuild switch` re-activates it and the fresh-deploy branch (`:471-479`, which never calls `warmsnap`)
 does restore live keycloak — but the foreign meta remains, so it re-wedges on the next due upgrade. Remediation
 is to remove the foreign `snapshot/`, not to redeploy."* The **precondition itself is unchanged and still
 correct**; only the mechanism/remediation sentences are wrong.
 ### A-redfix-3 [adversary] — **CLOSED 2026-07-09T08:59Z** (accepted + evidence retracted at `f64d102`; STATUS:28-29,564-572 re-read and correct) — STATUS's `main.go` "present in both clones ⇒ written outside git" evidence is void: `/srv/cc-ci` is a **symlink** to `/srv/cc-ci-orch`, so there is one file, not two
 Severity **INFO**. No VETO, no gate impact, no DoD impact. The *conclusion* (untracked, not from this phase,
 leave it alone) is fine; the supporting evidence is an artifact.
 **What STATUS-redfix.md (`42bc4e4`, lines ~470-472) claims:**
 > "`/srv/cc-ci-orch/cc-ci/main.go` **and** `/srv/cc-ci/cc-ci/main.go` — present in **both** clones, identical
 > size, identical mtime … i.e. written by something outside git (**a git operation cannot leave the same
 > untracked file in two clones**)."
 **Repro:**
    ls -la /srv/                                 # cc-ci -> /srv/cc-ci-orch   (symlink)
    stat -c '%d:%i  %n' /srv/cc-ci-orch/cc-ci/main.go /srv/cc-ci/cc-ci/main.go
    # -> 2049:3254604 for BOTH; stat -c 'links=%h' -> links=1
 Same device, same inode, link count 1: **one file, one clone, seen through a symlink.** The same holds for the
 Adversary clones (`/srv/cc-ci-orch/cc-ci-adv` and `/srv/cc-ci/cc-ci-adv` are inode `3206945`). The "two clones"
 observation carries **zero** information about the writer, and the identical size/mtime are tautological.
 **Risk bounded (read-only):** contents are a 281-byte hello-world `net/http` listener; `sha256
 bdbc3bf167cd20f30c00880005f4f994f17f3722660973c9c65c7bf33e81ffaf`; no `go.mod`; **`go` is not installed**;
 **nothing is listening on `:8080`**; `git log --all -- main.go` is empty; not gitignored; unreferenced by any
 tracked file. It cannot execute. Present in the **Builder's** clone only — **absent from both Adversary
 clone paths**, which is consistent with a single stray write into one working tree.
 **Ask of the Builder:** drop the "both clones ⇒ outside git" inference (keep the file, keep the flag). Agreed
 on not deleting it: neither of us created it.
 ---
 ### A-redfix-4 [adversary] — **CLOSED 2026-07-09T08:59Z** — all three asks applied at `e356698` and cold-re-verified by me at wake #45 (probes deleted; oscillation + silence recorded; precondition re-anchored to "never deploy `07fc6d4`"). The underlying *behaviour* remains open under **B-redfix-5**, which is deferred, not fixed. — the B-redfix-5 wedge has a **weekly autonomous re-trigger** (`nightly-sweep.timer`) and fails **silently**: post-arming, keycloak oscillates down-a-week / up-a-week with `nightly-sweep.service` reporting `Result=success`
 Severity **MED** (operator-facing; describes a *deferred* item's blast profile, **not** a DoD item).
 **No VETO. Does not reopen the phase.** M1 + M2 PASS stand. Merge target `b5f2b10` unchanged and still correct.
 Supersedes the "absence of a re-trigger" half of **A-redfix-2** (my own error) and, with it, the text the
 Builder accepted into STATUS-redfix.md at `f64d102`.
 **Both prior descriptions of the wedge are wrong, in opposite directions.**
 - Builder (`68b51d5`): "no code path redeploys it. **Recovery is manual.**" → wrong; it recovers unattended.
 - Me (`b358b7e`, A-redfix-2): "no re-trigger … heals on the next **reboot / `nixos-rebuild switch`**" → wrong;
  it re-triggers **weekly**, and an unrelated `switch` does *not* heal it.
 **The re-trigger.** `warm_reconcile.py` has **two** invokers, not one. The second is the sweep:
    nightly-sweep.timer   OnCalendar=Sun *-*-* 03:00:00, Persistent=true
      └─ nightly-sweep.service → /etc/cc-ci/runner/nightly_sweep.py   (CCCI_REPO=/etc/cc-ci)
           └─ main():147 roll_warm_infra()          # unconditional after the _another_run_active() guard
                └─ for app in WARM_APPS = ["keycloak", "traefik"]:   # nightly_sweep.py:39,57
                     subprocess.run([python, warm_reconcile.py, app])   # nightly_sweep.py:59-61
 **Observational proof that the sweep — not `warm-keycloak.service` — reconciles keycloak** (cc-ci, cold):
    systemctl show warm-keycloak.service -p ExecMainStartTimestamp   # → Wed 2026-06-17 17:29:31 UTC (once)
    stat -c %y /var/lib/ci-warm/keycloak/last_good                   # → 2026-07-05 03:04:51.209 +0000
    systemctl list-timers --all | grep nightly-sweep                 # → last trigger Sun 2026-07-05 03:04:50
 `write_last_good()` is called **only** from `reconcile()`. The slot file was written **18 days after** the unit
 last ran, and **1.2 s after** the sweep fired. ⇒ `reconcile("keycloak")` ran from the sweep. (`journalctl -u
 nightly-sweep.service` retains 1 line — the empty grep is *retention*, not absence. Do not read it as evidence.)
 **Why both of our probes missed it — the tell.** The invoking timer's name never contains the app name:
    systemctl list-timers --all | grep -i keycloak        # → EMPTY, yet a timer DOES drive it
    git grep -n warm-keycloak -- nix/ | grep -iE 'timer|Restart='   # → EMPTY, yet a timer DOES drive it
 Both are **unfalsifiable by construction**. STATUS-redfix.md (`f64d102`) currently prescribes *both* as the
 verification commands for "no re-trigger". They must be removed. The sound probe is to grep for the **callee**:
    git grep -n "warm_reconcile" -- runner/ nix/ | grep -v "^runner/warm_reconcile.py"
 Same class as the `e3b0c44298fc1c14` empty-input tell (B-redfix-8) and the `restore()`/`docker` ordering tell:
 a probe that returns the "all clear" value for a reason unrelated to the property under test.
 **Corrected dynamics** (only once armed — i.e. only if a foreign `snapshot/meta.json` sits in the slot **and**
 an upgrade is pending, `current != latest`):
 | sweep | state on entry | branch taken | outcome |
 |---|---|---|---|
 | N   | deployed, upgrade pending | stateful upgrade `:511-514` → `abra.undeploy` → `snapshot()` → guard raises | **keycloak DOWN** |
 | N+1 | not deployed | fresh-deploy `:471-479` → redeploys `current`; **never calls `warmsnap`** | keycloak UP (old version) |
 | N+2 | deployed, upgrade still pending | stateful upgrade again → guard raises | **keycloak DOWN** |
 ⇒ **keycloak alternates DOWN one week / UP the next**, indefinitely, until the foreign `snapshot/` is deleted
 or `current == latest`. Not a one-shot outage, and not manual-recovery-required.
 **And it is silent.** `roll_warm_infra()` captures the subprocess rc and **ignores it** (`nightly_sweep.py:59-63`
 — `print(f"nightly: reconcile {app} rc={rc}")`, no raise), so a wedged reconcile does **not** fail
 `nightly-sweep.service` (`Result=success`). The guard raises at `:514`, which is **outside** the try/except at
 `:520-525` (that is B-redfix-5) and **upstream of every `write_alert()`** — so **no alert is written**. The only
 trace is a journal line `nightly: reconcile keycloak rc=1`, in a unit whose output is not retained.
 **A weekly production outage of the shared SSO provider would surface nowhere.**
 **Good news — the trap cannot arm itself.** Arming requires the canonical seed to run from a tree that has the
 keycloak enrollment but **not** the fix. The enrollment is **branch-only**:
    git show origin/main:tests/keycloak/recipe_meta.py | grep WARM_CANONICAL   # → False   (= deployed /etc/cc-ci)
    git show 07fc6d4:tests/keycloak/recipe_meta.py     | grep WARM_CANONICAL   # → True    (enrollment, NO fix)
    git show b5f2b10:tests/keycloak/recipe_meta.py     | grep WARM_CANONICAL   # → True    (enrollment + fix)
 The deployed checkout `/etc/cc-ci` carries `WARM_CANONICAL = False`, so **no autonomous node activity — including
 Sunday's sweep — can write foreign meta into `/var/lib/ci-warm/keycloak/`.** Verified: the slot holds only
 `last_good` (no `snapshot/`, no `canon-*`), and keycloak is currently deployed and healthy (`_app` + `_db`).
 **⇒ The precondition is self-maintaining, and its landmark in STATUS is wrong.** It is not "the slot must be clean
 *at merge time*" (a `git merge` executes nothing). The binding statement is:
 > **Never deploy `07fc6d4`** — or any tree carrying the enrollment without `canonical_ns()`. That intermediate
 > state is the *only* thing that can arm the trap. Merging `b5f2b10` ships enrollment **and** fix together, so
 > canonical seeds land in `canon-keycloak/` and the slot never goes foreign.
 **Secondary correction (`nixos-rebuild switch`).** `warm-keycloak.service`'s `ExecStart` embeds the runner by
 store path — `/nix/store/…-runner/warm_reconcile.py`, and `warmsnap.py` lives in that **same** derivation
 (`…-runner/harness/warmsnap.py`). So a switch restarts the unit **iff `runner/**` changed**. A switch that
 touches anything else (e.g. sshd config) leaves the store path identical and does **not** re-run reconcile —
 so "heals on `nixos-rebuild switch`" is false in general. Deploying the merge *does* change `runner/**`, which
 means **the merge's own activation switch is itself a reconcile trigger**, alongside the Sunday sweep.
 **Asks of the Builder (STATUS-redfix.md text only — no code change, nothing reopens):**
 1. Delete both `grep -i keycloak` / `git grep warm-keycloak -- nix/` "no re-trigger" probes; they cannot fail.
 2. Replace "recovery is manual" **and** "heals on reboot / `nixos-rebuild switch`" with the weekly
   down/up oscillation above, and state that it is **silent** (`Result=success`, no `write_alert`).
 3. Re-anchor the merge precondition from "slot clean at merge time" to "**never deploy `07fc6d4`**".
 Remediation if ever armed is unchanged and still correct: **delete the foreign `snapshot/` from the slot**;
 do not redeploy keycloak (the sweep does that for you, and it is what masks the fault).
 ---
 ### A-redfix-1 — ADDENDUM (wake #45, 2026-07-09T08:59Z): the embedded value is the **same live credential as B-redfix-8**, confirmed at value level; and rotation will **not** clean this file
 Still **OPEN**. Re-verified cold on cc-ci this wake, without printing the secret:
    ssh cc-ci 'stat -c "%a %U:%G" /etc/cc-ci/.git/config'                     # → 644 root:root
    ssh cc-ci 'grep -oE "https://[^/@]+:[^/@]+@" /etc/cc-ci/.git/config \
                 | sed -E "s#https://[^:]+:##; s#@$##" | tr -d "\n" | sha256sum | cut -c1-16'
 **EXPECTED / observed: `3fcea78925015fc9`** — byte-identical to the B-redfix-8 rotation sentinel
 (`sha256(GITEA_PASSWORD)[:16]`, re-confirmed this wake from `/srv/cc-ci/.testenv` on the **orchestrator**).
 So A-redfix-1 and B-redfix-8 are **two exposures of one still-unrotated, push-capable credential**, and the
 sentinel matching proves it is **still not rotated**. (Extract-then-hash, never echo. `python3` is absent on
 cc-ci — do not run the orchestrator's python probe over `ssh`; that is the `e3b0c44298fc1c14` empty-input tell.)
 **New, and operator-relevant: rotation does not fix this file, and may recreate the exposure.**
 `/etc/cc-ci` is a **manual `git clone`**, not nix-generated — `nix/hosts/cc-ci-hetzner/configuration.nix:7`
 documents `git clone --recursive https://git.autonomic.zone/recipe-maintainers/cc-ci.git /etc/cc-ci`. Nothing
 regenerates `.git/config` on `nixos-rebuild switch`. Therefore:
 - After rotating `GITEA_PASSWORD`, the **old** value persists verbatim in this 0644 file (inert, but it should
  still be scrubbed — it is the value published at `14c7dee`).
 - If the operator re-clones or re-embeds the **new** password in the remote URL, the 0644 exposure **returns**.
 **Recommendation (operator, alongside the B-redfix-8 rotation):** point the remote at a credential-less URL
 (`https://git.autonomic.zone/recipe-maintainers/cc-ci.git`) and authenticate via a `credential.helper` /
 token file with `0600`, or at minimum `chmod 0600 /etc/cc-ci/.git/config`. Do not carry userinfo in the URL.
 **Rotation blast radius is small** (checked, so this is not a reason to delay): `GITEA_PASSWORD` is consumed
 only by `scripts/bootstrap-drone-oauth.sh` (a one-off bootstrap). `scripts/recipe-mirror-sync.sh` pushes with
 an **OAuth token**, not the password, so the weekly sweep's mirror sync does **not** depend on it.
 **CORRECTION (wake #55): the "small blast radius" reasoning is HALF WRONG, and the OAuth token is ALSO
 exposed.** The above argues rotation is safe because mirror-sync uses a token, not the password — but that
 same `oauth2` token (`recipe-mirror-sync.sh:39`, `https://oauth2:${TOKEN}@…`, PUSH-capable) is itself leaked
 in **117** world-readable `.git/config` copies under `/var/lib/cc-ci-runs` (sentinel `9c44a1aea2ecb389`;
 verified live: `GET /api/v1/user` → 200 `autonomic-bot`/id 64). So there are **two** live push-capable
 credentials to rotate, and rotating `GITEA_PASSWORD` alone leaves the token exposed. The token rotation DOES
 have a real dependent (mirror-sync) — so it must be re-minted and the new value delivered to mirror-sync's
 secret source, not merely revoked. See STATUS-redfix.md steps 3–4 (updated) for the combined remedy.
 ### A-redfix-1 — ADDENDUM (wake #53, 2026-07-09T10:15Z): the "sole copy" claim is FALSIFIED — **78** world-readable copies, root-caused to a git `insteadOf` rewrite that regenerates them on every CI run
 The original finding asserted (see above, line ~47): "`/etc/cc-ci` is the **sole** copy whose parents are
 world-traversable." I disbelieved that universal and probed it cold on cc-ci. **It is false.** Enumerating
 every cred-bearing `.git/config` and testing each for uid-1000 readability (extract-then-hash, never echo):
    ssh cc-ci 'find / -xdev -name config -path "*/.git/*" 2>/dev/null | while read f; do
      grep -q "autonomic-bot:" "$f" || continue
      setpriv --reuid=1000 --regid=1000 --clear-groups cat "$f" >/dev/null 2>&1 && echo "$f"; done'
    # → 78 world-readable copies: 68 under /var/lib/cc-ci-runs/*/abra/recipes/*/.git/config,
    #   8 in /nix/store/*-source/.git/config (0444, read-only fs), /tmp/v/.git/config, /etc/cc-ci/.git/config
 All 78 carry the **same** live credential — `sha256(pw)[:16] = 3fcea78925015fc9`, identical to the B-redfix-8
 sentinel (empty-input control `e3b0c44298fc1c14` to prove the extractor fires). So the exposure is **78×**, not
 1×, and `/etc/cc-ci` is not special.
 **Root cause (grep the callee, not the value).** `run_recipe_ci.py:360` clones a **clean, credential-less**
 URL (`https://git.autonomic.zone/{src}.git`). The userinfo is injected by a global git rewrite in
 `/root/.gitconfig`:
    url.https://autonomic-bot:<pw>@git.autonomic.zone/.insteadOf = https://git.autonomic.zone/
 Every per-run recipe clone the harness makes therefore bakes the cleartext password into its `.git/config` at
 mode **0644**, under `/var/lib/cc-ci-runs/<rid>/abra/recipes/<recipe>/` (world-traversable parents). This is
 the **generator**: the 68 run-dir copies accrete one-per-recipe-per-run and **regenerate autonomously on the
 next CI run** — not just on a manual re-clone as the prior addendum supposed.
 **Consequence for the remedy.** The prior recommendation (`chmod 0600 /etc/cc-ci/.git/config`, or delete
 `/etc/cc-ci`) fixes **1 of 78** and does nothing about regeneration. A durable operator fix must remove the
 credential from git's URL layer, e.g.:
 - replace the `insteadOf`-with-userinfo rewrite in `/root/.gitconfig` with a `credential.helper` (token in a
  `0600` file), so harness clones carry **no** userinfo; and
 - scrub the existing run-dir + `/etc/cc-ci` + `/tmp/v` copies (the `/nix/store` ones are 0444 on a read-only
  store and clear on GC).
 Otherwise the next `!testme`/nightly-sweep run re-exposes the (even freshly-rotated) value at 0644.
 **Scope note — NOT a gate, NOT a VETO, DONE stands.** The dashboard "no secrets" invariant was independently
 **re-verified holding** this wake: `/runs/<rid>/abra/recipes/<recipe>/.git/config` → **404** on the live
 dashboard (blocked by the `len(parts)==2` guard + the `_RUN_FILES` allow-list; positive control
 `/runs/985/results.json` → 200, `has_cred=False`). This exposure is **host-local filesystem only** (any local
 uid, incl. containers sharing the host mount), which is squarely the operator-only A-redfix-1/B-redfix-8
 territory. It **widens** their remediation scope; it does not reopen any D-gate or DoD item.
 ### A-redfix-1 — ADDENDUM (wake #55, 2026-07-09T10:4xZ): wake-#53 `insteadOf` root cause RETRACTED; real generator = the sweep's `CCCI_SKIP_FETCH` copytree; a SECOND credential (oauth2 token) also exposed
 Adjudicated the Builder's wake-#54 rebuttal cold. Corrections to my own wake-#53 addendum above:
 - **RETRACT the `insteadOf` root cause.** Reproduced first-hand on the node (git 2.47.2): a clone
  whose transport `url.<t>.insteadOf`-rewrites still stores the **original** URL; `insteadOf` injects
  no userinfo when the clone URL is credential-less. My "regenerates via insteadOf" was wrong.
 - **Real generator:** canonical `/root/.abra/recipes/*/.git/config` carry the cred (safe at rest under
  `/root` = `0700`); `run_recipe_ci.py:348-353` `CCCI_SKIP_FETCH=1` `copytree`s them into
  `/var/lib/cc-ci-runs` (`0755`), dropping the protection.
 - **The exposure IS autonomously regenerated — weekly, by the sweep** (correcting the Builder's
  "manual-* = hand-runs" claim). `run_id()`→`"manual"` for any non-Drone run; `nightly_sweep.py:88`
  runs `run_recipe_ci.py` outside Drone with `CCCI_SKIP_FETCH=1`, so the sweep's runs are the
  `manual-<pid>` dirs. Freshest copies dated 2026-07-05 03:37–03:59Z = the sweep's last fire.
 - **SECOND credential found (Builder census missed it).** `/var/lib/cc-ci-runs` holds **123**
  world-readable cred-bearing configs: **68** `autonomic-bot/3fcea78925015fc9` (B-redfix-8 password)
  **+ 55** `oauth2/9c44a1aea2ecb389` (a distinct OAuth2 token, sourced from bluesky-pds,
  custom-html-tiny, gitea, mumble). Remedy must scrub/rotate **both**.
 Still **OPEN**, still **operator-scope**, **no gate / no DoD impact, no VETO**. Endorse the Builder's
 proposed remedy (strip userinfo from canonical origins + scrub + `chmod 0750 /var/lib/cc-ci-runs`)
 with the two amendments above (both credentials; canonical-origin fix is required because a one-time
 scrub is re-armed by the next sweep). Full reasoning + reproductions in REVIEW-redfix.md wake #55.
 ### B-redfix-8 — ADDENDUM (wake #58, 2026-07-09): exposure is **two published commits**, not one; and the `oauth2` token is **not** in git history
 Triggered by the Adversary's wake-#57 break-it probe (`5431252`), which verified the token does not leak to
 the dashboard / weekly reports / Drone log API — but did **not** probe the **public git mirror**, which is the
 surface B-redfix-8 is actually about. Closing that gap surfaced a defect in my own STATUS.
 **Method.** Exhaustive scan of the whole object db (`git cat-file --batch-all-objects` → **3099 blobs**),
 literal-value match, `awk`-free (no `awk` on the orchestrator; an `awk`-based pipeline silently yields an
 empty blob list and a **vacuous** `0 hits`). Token pulled root-only from a world-readable run-dir
 `.git/config` into a `chmod 600` file, verified `sha256[:16]=9c44a1aea2ecb389` (matches the Adversary's
 wake-#55 sentinel), `shred`-removed after. Positive control: the known-leaked password **must** be found.
 **Results.**
 - **password** (`sha16 3fcea78925015fc9`) → **2** blobs / **2** published commits:
  `14c7dee`(blob `fd21fcb8`, 33408 B) and `223cc16`(blob `bcc31b55`, 34353 B). Both are ancestors of
  `origin/main` ⇒ both mirror-served. Neither reachable from HEAD. `e99e2b3` and HEAD are clean.
  ⇒ STATUS's "**only** the historical commit `14c7dee` serves it" was **wrong and understated the exposure**.
  A history scrub targeting `14c7dee` alone would leave `223cc16` serving the live credential. STATUS corrected.
 - **oauth2 token** (`sha16 9c44a1aea2ecb389`) → **0** of 3099 blobs. The 271 blobs matching the *string*
  `oauth2:` are `recipe-mirror-sync.sh:39` `https://oauth2:${TOKEN}@…` variable interpolation and machine-docs
  prose citing the sha16 sentinel — no literal value.
 **Consequence for the operator remedy.** The two credentials are **not symmetric**:
 - **password** — exposed on the **public mirror** (permanent, unauthenticated, in every clone) **and** on the
  filesystem. Rotation is urgent; scrubbing must cover **both** commits, and cannot recall existing clones.
 - **oauth2 token** — exposed on the **filesystem only** (world-readable `.git/config` copies, A-redfix-1),
  regenerated weekly by the sweep. Rotation + `chmod`/userinfo-strip suffices; no history rewrite needed.
 **Guard for future probes.** Any scan of this repo reporting **0** password-bearing blobs is a **broken probe**
 (missing `awk`, empty sentinel `e3b0c44298fc1c14`, or names-only grep of tracked paths). The correct answer
 is **2**. Always run the positive control.
 Still **operator-scope, outside DoD, no gate impact.** `## DONE` stands; no VETO.
 ### B-redfix-8 — ADDENDUM 2 (wake #58b, 2026-07-09): blob-count figure was brittle; the invariant is 2-and-0, not a total
 The Adversary's wake-#58 adjudication (`63f7001`) independently CONFIRMED both results (token `0` blobs,
 password exactly `2` — `fd21fcb8@14c7dee` + `bcc31b55@223cc16`, both mirror-served ancestors) — but reported
 its total as **3061 blobs** where I reported **3099**. Neither is wrong; the *total* is simply not an invariant.
 Root-caused, three independent sources of drift:
 1. **New commits add blobs.** Same host now reads **3104** (my wake-#58 commits landed after the scan).
 2. **`--batch-all-objects` counts unreachable objects** that `git rev-list --all` omits — **49** of them here.
   Reachable-only is **3026**.
 3. **`git clone /local/path` hardlinks the entire object store**, so a *local* clone inherits those unreachable
   objects (verified: my cold clone also reads 3104). A clone from the **remote** gets only reachable objects —
   which is why the Adversary's number is lower, and why it is the right scope for a *mirror-exposure* question.
 **Consequence.** My STATUS quoted `# → 3099 lines` as the expected output of the documented reproduction. A
 future agent running that command will see a different number, and could read the mismatch as either a broken
 probe or a changed repo — both wrong. STATUS corrected: the reproduction now asserts `wc -l > 100` as a sanity
 floor and names the **durable invariants — password ⇒ exactly 2 blobs, token ⇒ exactly 0** — which held across
 all four scans (3099 / 3104 / 3026 / 3061) precisely because they are scope-independent.
 Lesson, same shape as the `awk` near-miss: **an incidental figure recorded as if it were an assertion becomes a
 future false alarm.** Record the invariant, not the measurement.
 No new exposure, no gate impact, operator-scope. `## DONE` stands; no VETO.
--- a/machine-docs/DECISIONS.md
+++ b/machine-docs/DECISIONS.md
@ -1609,52 +1609,3 @@ trim job observed → adequate; revisit only if a cap is ever needed.
  `dashboard/dashboard.py` + `tests/unit/test_dashboard.py` would be reformatted — confirmed pre-existing
  at HEAD f68f1c5, outside the settings diff. Flagged for the dashboard owner / orchestrator; not fixed
  here (narrow scope).
 ## Warm state is keyed by STACK NAMESPACE, not by recipe (phase redfix, F-redfix-4, 2026-07-09)
 **Settled.** `/var/lib/ci-warm/<ns>/` is a slot owned by exactly ONE deployed stack. `ns` comes from
 `canonical.canonical_ns(recipe)` for the data-warm canonical and `warmsnap.live_slot(recipe)` for the
 live-warm reconciler. A recipe that is both (only `keycloak` today, via `warm.WARM_DOMAINS`) gets
 `canon-<recipe>` for its canonical, so the two never share a slot. The canonical's **domain and its slot
 derive from the same `canonical_ns()`**, so they cannot drift apart — that coupling is the invariant, not
 the `canon-` string. Every slot `<ns>` maps 1:1 to the stack at `warm.stable_domain(<ns>)`.
 Rejected alternatives:
 - **Skip `seed_canonical` for `WARM_DOMAINS` recipes.** Silently de-enrolls keycloak and re-opens the DoD
  item ("keycloak enrolled … verified green"). The Adversary named this as not acceptable; agreed.
 - **Key every slot by the full domain / stack name** (the "purest" model). Correct, but it relocates the
  reconciler's `last_good` for keycloak+traefik and `canonical.json` + `snapshot/` for 15 existing
  canonicals on a live node, requiring a migration shim for a phase whose mandate is to fix red, not
  re-architect. The chosen scheme is the same model applied only where two stacks actually contend, and
  needs no migration (keycloak's canonical had never been seeded).
 - **Guard-only (`_assert_slot_not_foreign` alone).** Turns silent destruction into a loud failure but leaves
  keycloak's canonical unable to ever seed. Kept as defence in depth BEHIND the slot separation, not as the
  fix.
 Consequence for future enrollments: adding a recipe to `warm.WARM_DOMAINS` automatically namespaces its
 canonical on both axes. `prune_stale()`'s "reconciler dirs are never pruned" invariant is now structural
 (a `<recipe>/` dir never gains a `canonical.json`) rather than an incidental property that de-enrollment
 could falsify.
 ## 2026-07-10 — Gitea credential rotation + git auth moved to SSH (operator-directed)
 Both leaked `autonomic-bot` credentials were rotated and the exposures scrubbed (was B-redfix-8 /
 A-redfix-1 / B-redfix-9, previously operator-scope):
 - **Account password** rotated (old value now returns HTTP 401). It had been world-readable on the
  public mirror in commits `14c7dee` + `223cc16`. History was rewritten (`git filter-repo --replace-text`)
  to redact it and force-pushed; `main` went `bfa5396 -> 0d8adba` (tree byte-identical, history-only).
  All clones + node `/etc/cc-ci` were reset to the rewritten `main`. Backup of pre-rewrite repo saved at
  `~/cc-ci-backups/pre-scrub-*` (mirror + bundle). **Caveat:** the orphaned commits are still fetchable
  by exact SHA on the public mirror until git.autonomic.zone runs a site-admin GC (value is dead, so inert).
 - **`bridge_gitea_token`** (the oauth2 token, sops `secrets.yaml`) rotated: fresh PAT minted, re-encrypted,
  deployed via `nixos-rebuild ?submodules=1`, old token + all 8 other stale PATs revoked. Only one PAT
  remains: `cc-ci-bridge-*` (id 36).
 **Git auth is now SSH, not HTTP.** git.autonomic.zone's Gitea only honours *preemptive* HTTP Basic auth;
 credential-helper (challenge-response) always 401s on private repos — do not rely on it. Remotes are SSH
 (`git@git.autonomic.zone:...`, port 2222 via `~/.ssh/config`); dedicated keys `cc-ci-orchestrator-20260710`
 (orchestrator, `~/.ssh/autonomic-bot-gitea-ed25519`) and `cc-ci-node-20260710` (node, `/root/.ssh/...`).
 `.gitmodules` secrets URL is SSH. `recipe-mirror-sync.sh` pushes over SSH (no token written into recipe
 `.git/config` -> fixes the copytree leak). The bot **password stays in `/srv/cc-ci/.testenv` for API use only**
 (comments / PR / repo creation); the weekly Thursday upgrade run picks it up by sourcing `.testenv`.
 Node run-tree `/var/lib/cc-ci-runs` chmod 0750; 144 world-readable cred-bearing config copies scrubbed.
--- a/machine-docs/DEFERRED.md
+++ b/machine-docs/DEFERRED.md
@ -427,29 +427,3 @@ reachable via the operator/dev STAGES escape — production drone runs always ru
  key on stdout (write to a run-scoped sidecar the test reads), or register the minted key in the harness
  redaction set so even the RAW log is scrubbed. Low priority (RAW log is access-controlled; key is ephemeral).
 - **Filed by:** Builder, phase prevb (acknowledging Adversary [F-prevb-C]).
 ## B-redfix-5 — warm reconciler's rollback `restore()` is outside the upgrade's `try/except`
 - **What:** in `runner/warm_reconcile.py`, **two** post-`abra.undeploy()` warmsnap calls sit OUTSIDE the
  `try/except` guarding the upgrade — **(a)** the upgrade path `:512-514` (`undeploy` → `wait_undeployed` →
  `warmsnap.snapshot(...)`) and **(b)** the unhealthy-rollback path `:534-536` (`undeploy` →
  `wait_undeployed` → `warmsnap.restore(...)` → `deploy_version(last_good)` at `:537`). If either raises for
  any reason (absent/corrupt snapshot, foreign slot, docker error, "no volumes found") the exception
  propagates, the following `deploy_version` never runs, and live keycloak is left **undeployed**, taking
  down the shared OIDC provider `lasuite-*`/`drone` depend on. Site (a) fires on the *normal* upgrade path —
  no rollback required.
 - **Why deferred, not fixed:** structural gap, present at `07fc6d4` and predating the keycloak enrollment.
  Not part of F-redfix-4's published clearing condition; the Adversary explicitly concurred it is **not a
  VETO** and was "correctly filed rather than silently fixed".
 - **Reachability, corrected (wake #42).** The earlier text here said F-redfix-4 "supplied the only
  *reachable* trigger and that is now closed". That is wrong in one direction: `b5f2b10` *adds* a raise path
  — `_assert_slot_not_foreign()` is called inside BOTH `warmsnap.snapshot()` (`:158`) and `warmsnap.restore()`
  (`:209`). What makes the composition unreachable **today** is that cc-ci's live slot
  `/var/lib/ci-warm/keycloak/` holds no `snapshot/meta.json` at all (`read_meta` → `None`; an unclaimed slot
  is free to claim) — not the closure of F-redfix-4. The one state that writes a foreign domain into the
  live slot is a **pre-fix (`07fc6d4`) canonical seed**. Recorded as a MERGE PRECONDITION in STATUS-redfix.md.
 - **Needed from operator:** decide the safe behaviour for a DB-backed app after a forward migration —
  (a) redeploy `last_good` anyway even though data was not restored, or (b) die loudly + leave a breadcrumb
  rather than start on unrestored data. That trade-off is a real safety call, not a mechanical fix, which is
  why it was not settled inside a remediation commit.
 - **Filed by:** Builder, phase redfix (from Adversary F-redfix-4 consequence 3).
--- a/machine-docs/JOURNAL-redfix.md
+++ b/machine-docs/JOURNAL-redfix.md
--- a/machine-docs/REVIEW-redfix.md
+++ b/machine-docs/REVIEW-redfix.md
--- a/machine-docs/STATUS-redfix.md
+++ b/machine-docs/STATUS-redfix.md
@ -7,456 +7,7 @@ gitea, keycloak) → isolate → root-cause → classify (flake vs genuine; reci
 warm-machinery vs load) → FIX each (recipe PR or harness improvement) → verify green. No standing
 exceptions. Nothing merged.
-## DONE — 2026-07-09T00:18Z
+## DONE — 2026-06-18T07:09Z
 Phase `redfix` COMPLETE. All six canon-sweep failures investigated in isolation, root-caused, classified,
 **FIXED — each via a recipe PR or a harness improvement — and verified green**; no recipe left as a standing
 exception; nothing merged (operator merges). Every gate has a fresh Adversary PASS in REVIEW-redfix.md and
 there is **no standing VETO**:
 - **M1 PASS** @ 2026-06-18T01:18Z (investigation/classification cold-verified).
 - **M2 PASS** @ 2026-07-09T00:18Z (F-redfix-4 remedy cold-verified; VETO CLEARED; supersedes the
  2026-06-18T07:06Z M2 PASS, which was given against parent `07fc6d4`).
 Adversary findings F-redfix-1/2/3/4 are all **CLOSED**; no open blocking finding.
 The four `A-redfix-*` findings are also non-blocking and are accounted for here, so nothing dangles:
 - **A-redfix-1** (`/etc/cc-ci/.git/config` is 0644 and embeds the Gitea bot password) — **OPEN**, out of redfix
  scope, pre-existing undeclared server state, no DoD item. Mirrors my B-redfix-7. Not created by this phase.
  **It is the SAME credential as B-redfix-8, not merely a similar one** — verified first-hand 2026-07-09T09:1xZ:
  the password extracted from the `.git/config` remote URL and the `GITEA_PASSWORD` in the orchestrator's
  `/srv/cc-ci/.testenv` both hash to `3fcea78925015fc9` (neither is the empty-string artifact
  `e3b0c44298fc1c14`). **Two exposures, one credential, still unrotated.** Folded into the B-redfix-8 remedy
  below, because rotating the password does **not** scrub this file.
 - **A-redfix-2** (the B-redfix-5 wedge's self-heal mechanism) — **PARTIALLY WITHDRAWN by the Adversary at
  wake #44.** Its first half stands (a code path *does* redeploy keycloak — the fresh-deploy branch). Its
  second half ("no re-trigger; heals on reboot") is **false** and is superseded by A-redfix-4.
 - **A-redfix-3** (the `main.go` "both clones" evidence) — **ACCEPTED**; that evidence is retracted below.
  `/srv/cc-ci` is a symlink to `/srv/cc-ci-orch`. The file's origin remains unexplained; risk inert.
 - **A-redfix-4** (a weekly `nightly-sweep.timer` *does* re-trigger reconcile; the wedge is **silent** and
  **recurring**) — **ACCEPTED and independently re-verified against the node**, not taken on authority. The
  B-redfix-5 precondition below is rewritten accordingly: two unfalsifiable probes deleted, the wedge
  described as a weekly alternating DOWN/UP outage that raises no alert, and the precondition re-anchored
  from "slot clean at merge time" to **"never deploy `07fc6d4`"**. The trap is **not armed** and **cannot be
  armed by autonomous node activity** (the tree the sweep runs has `WARM_CANONICAL = False`).
  **MECHANISM CORRECTED (wake #54; Adversary REVIEW wake #54, re-verified first-hand by the Builder).** The
  sweep does **not** run `origin/main`. `nightly-sweep.service` ExecStart is a nix-store wrapper that sets
  `CCCI_REPO=/etc/cc-ci` and execs `cc-ci-run "$CCCI_REPO/runner/nightly_sweep.py"`, so the tree that
  executes is the **deployed checkout `/etc/cc-ci`** (branch `main` @ `d11f8f5`, clean). The guarantee is a
  property of **what is deployed to `/etc/cc-ci`**, so a future **deploy** — not a merge to `origin/main` —
  is what could arm the wedge. Conclusion unchanged (trap disarmed); only the named tree was wrong. Verify:
      ssh cc-ci 'git -C /etc/cc-ci rev-parse --abbrev-ref HEAD; grep -n "^WARM_CANONICAL" /etc/cc-ci/tests/keycloak/recipe_meta.py;
        git -C /etc/cc-ci merge-base --is-ancestor 07fc6d4 HEAD && echo ARMED || echo disarmed'
      # EXPECTED: main / 16:WARM_CANONICAL = False / disarmed   (a real assignment, not the line-15 comment)
 All four corrected operator-facing *descriptions*, not DoD items. No VETO at wake #43 (`b358b7e`) or wake #44
 (`84a1666`, 2026-07-09T08:47Z). **Note on provenance:** A-redfix-2's false half entered this file at `f64d102`
 because I adopted the Adversary's stated mechanism *and its two probe commands* without re-deriving them. The
 probes returned the all-clear for a reason unrelated to the property under test (they grep the app name;
 the timer is named `nightly-sweep`). Both are now deleted, and every claim in the rewritten block below was
 re-checked first-hand.
 ## ⚠ OPERATOR ACTION REQUIRED — B-redfix-8 (HIGH, open, outside DoD)
 Phase DoD is met, but **one HIGH item is open and only an operator can close it.** It does not block `##
 DONE` (no DoD item depends on it) and is recorded in full in BACKLOG-redfix.md.
 **WHAT.** The live Gitea bot password (`GITEA_PASSWORD` in `/srv/cc-ci/.testenv`) was committed to
 `machine-docs/BACKLOG-redfix.md`, pushed to `origin/main`, and is served **in cleartext to the
 unauthenticated public internet** by the public mirror. It is push-capable to `recipe-maintainers/*`. HEAD
 and the redaction commit `e99e2b3` are clean. Redaction cannot unpublish it — the value is permanent in
 history and in every clone.
 **CORRECTED (wake #58, 2026-07-09):** an earlier version of this entry said "**only** the historical commit
 `14c7dee` serves it." That **understated the exposure**. An exhaustive scan of every blob in the object db
 (`git cat-file --batch-all-objects`, literal-value match) finds the password in **two distinct
 blobs**, carried by **two published commits** — both ancestors of `origin/main`, so both are mirror-served:
 | commit | blob | `machine-docs/BACKLOG-redfix.md` size |
 |---|---|---|
 | `14c7dee` | `fd21fcb8ac02bff9917531c8175a29e450c41b19` | 33408 |
 | `223cc16` | `bcc31b55860f782056ba3ccdf13165191bd329b5` | 34353 |
 Neither blob is reachable from HEAD. Scrubbing only `14c7dee` would leave `223cc16` serving the credential.
 **WHERE.** `git.autonomic.zone/recipe-maintainers/cc-ci` →
 `/raw/commit/14c7dee/machine-docs/BACKLOG-redfix.md` **and**
 `/raw/commit/223cc16/machine-docs/BACKLOG-redfix.md`
 **HOW to confirm exposure** (unauthenticated fetch; a raw blob at a fixed sha is immutable, so served size
 must equal object size):
    git cat-file -s 14c7dee:machine-docs/BACKLOG-redfix.md    # → 33408
    git cat-file -s 223cc16:machine-docs/BACKLOG-redfix.md    # → 34353
 An unauthenticated fetch of **either** raw URL returning **HTTP 200** at the matching byte count is the
 leak, not an error page.
 **Exhaustive-scan reproduction** (names/counts only; never prints the value). The `awk`-free form matters —
 `awk` is absent on the orchestrator and an `awk`-based blob list yields an **empty** list, i.e. a vacuous
 `0 hits`:
    git cat-file --batch-all-objects --batch-check='%(objecttype) %(objectname)' \
      | grep '^blob ' | cut -d' ' -f2 > /tmp/blobs        # assert  wc -l > 100  before trusting
    # then literal-match each blob against a chmod-600 sentinel; positive control MUST return 2
 **Do not assert an exact total blob count** — it is host- and time-dependent, and is *not* the invariant.
 Observed legitimately: **3099** (Builder wake-#58 scan), **3104** (same host, after that wake's commits),
 **3026** (reachable-only), **3061** (Adversary's cold clone, wake #58). Three sources of drift: (a) new
 commits add blobs; (b) `--batch-all-objects` counts **unreachable** objects (49 here) that `git rev-list`
 omits; (c) `git clone /local/path` **hardlinks the whole object store**, so a local clone inherits those
 unreachable objects while a clone from the *remote* receives only reachable ones. All four scans agree on
 the only figures that matter.
 **Positive control is mandatory.** The durable invariants, stable across every scan scope above:
 **password ⇒ exactly 2 blobs; token ⇒ exactly 0.** Any scan of this repo that reports **0** password-bearing
 blobs is a **broken probe**, not a clean repo.
 **The `oauth2` token (the second exposed credential) does NOT appear in git history.** Same exhaustive scan,
 same run, literal match against the live token (`sha256[:16]=9c44a1aea2ecb389`, extracted root-only to a
 `chmod 600` file, then `shred`-removed): **0 blobs** — independently reproduced by the Adversary at wake #58
 against its own cold clone. The 271 history blobs matching the string
 `oauth2:` are `scripts/recipe-mirror-sync.sh:39`'s `https://oauth2:${TOKEN}@…` **variable interpolation**
 plus machine-docs **prose citing the sha16 sentinel** — no literal value. So the token's exposure is
 **filesystem-only** (A-redfix-1: world-readable `.git/config` copies); it is **not** on the public mirror.
 The password's exposure is **both** filesystem **and** public mirror. The two credentials differ in blast
 radius; the operator remedy must not treat them as symmetric.
 **HOW to confirm it is still the live credential** (commits to the value without republishing it).
 Run this **on the orchestrator** — `/srv/cc-ci/.testenv` lives there, **not** on `cc-ci`:
    python3 -c 'import hashlib,re;v=re.search(r"^GITEA_PASSWORD=(.*)$",open("/srv/cc-ci/.testenv").read(),re.M).group(1).strip().strip(chr(34)).strip(chr(39));print(hashlib.sha256(v.encode()).hexdigest()[:16])'
 **EXPECTED.** Prints `3fcea78925015fc9` while still unrotated. **A different digest ⇒ rotated ⇒ `14c7dee`
 is inert ⇒ close B-redfix-8** — *but only if the probe was sound.* Two known-bad probes yield a different
 digest without any rotation, and must NOT be used to close this item:
 - **`e3b0c44298fc1c14` is the empty-input tell, not a rotation.** Running the command over `ssh cc-ci`
  fails to open `.testenv`, hashes the empty string, and prints `sha256("")[:16]` = `e3b0c44298fc1c14`
  (check: `printf '' | sha256sum`). Fix the probe; do not close B-redfix-8.
 - **Do not conflate the three digests.** `3fcea78925015fc9` = sha256(credential *value*)[:16] — the only
  rotation sentinel. `1994fd8d…` = sha256(the whole served *blob*) — immutable, changes never.
  `e3b0c44298fc1c14` = sha256(empty) — a broken probe.
 Exposure confirmed at **value level** (Adversary, 2026-07-09T07:34Z): the currently-valid, push-capable
 password appears **verbatim** in the publicly served body, not merely "a blob that once held a secret."
 **REMEDY (operator).** Rotate the Gitea bot password, then update **both** places that hold it. The credential
 is a Class-A1 external infra input (plan §4.4) — the Builder must not rotate or invent it. History rewrite is
 not available to the Builder either (`--force` is forbidden by the standing rules), so rotation is the only
 action that actually revokes the exposure.
 1. Rotate the bot password in Gitea.
 2. Update `GITEA_PASSWORD` in `/srv/cc-ci/.testenv` **on the orchestrator**.
 3. **Rewrite `/etc/cc-ci/.git/config` on `cc-ci` (A-redfix-1) — rotation does NOT scrub it.** That file is
   `644 root:root` and embeds the same password in the `origin` URL's userinfo. `/etc/cc-ci` is a **manual
   `git clone`**, not nix-generated (`nix/hosts/cc-ci-hetzner/configuration.nix:7` documents the clone), so no
   `nixos-rebuild` regenerates or scrubs it. Strip the userinfo from the URL rather than re-embedding the new
   password — re-embedding recreates the exposure verbatim:
       ssh cc-ci 'git -C /etc/cc-ci remote set-url origin \
         https://git.autonomic.zone/recipe-maintainers/cc-ci.git && chmod 600 /etc/cc-ci/.git/config'
   **CORRECTION (wake #54→#55, Builder, 2026-07-09): `/etc/cc-ci/.git/config` is 1 of many world-readable
   copies of TWO live credentials — step 3 alone fixes 1.** The "sole copy" claim above is **withdrawn**.
   Scope re-derived first-hand at wake #55 (a full-FS, per-file census — my wake-#54 count of 78 was BOTH
   undercounted AND single-credential; Adversary flagged both, wake #55). Two distinct live credentials, each
   its own `sha256(pw)[:16]` sentinel (empty-input control `e3b0c44298fc1c14`):
   | credential | sentinel | world-readable copies (uid-1000-readable) |
   |---|---|---|
   | `autonomic-bot` **password** (B-redfix-8) | `3fcea78925015fc9` | 78 (68 run-dir + 8 `/nix/store` 0444 + `/tmp/v` + `/etc/cc-ci`) |
   | `oauth2` **Gitea token** (NEW, wake #55) | `9c44a1aea2ecb389` | 117 (`/var/lib/cc-ci-runs`), sourced from 20 canonical clones |
   Per-file under `/var/lib/cc-ci-runs`: **62 carry BOTH**, 16 password-only, 55 token-only → **133 distinct
   world-readable cred-bearing files** there. **The `oauth2` token is LIVE and PUSH-CAPABLE** — verified
   `GET /api/v1/user` → 200 `login=autonomic-bot id=64`, and `recipe-mirror-sync.sh:39` pushes with it
   (`https://oauth2:<TOKEN>@…`). So **both** secrets must be rotated, not just `GITEA_PASSWORD`. Re-census:
       ssh cc-ci 'find / -xdev -name config -path "*/.git/*" 2>/dev/null | while read f; do
         setpriv --reuid=1000 --regid=1000 --clear-groups cat "$f" >/dev/null 2>&1 || continue
         grep -qE "autonomic-bot:|oauth2:" "$f" && echo "$f"; done | wc -l'
       # EXPECTED: ~135 today (133 in /var/lib + /tmp/v + /etc/cc-ci); after remediation: 0 (ignoring 0444 /nix/store)
   Legacy list (password only), retained for the step-3 command below — the `origin` in these carries the
   password sentinel `3fcea78925015fc9`:
   **68** under `/var/lib/cc-ci-runs/manual-*/abra/recipes/*/`, **8** in `/nix/store` (0444, read-only, clears
   on GC), **1** `/tmp/v`, **1** `/etc/cc-ci`. Verify:
       ssh cc-ci 'find / -xdev -name config -path "*/.git/*" 2>/dev/null | while read f; do
         grep -q "autonomic-bot:" "$f" || continue
         setpriv --reuid=1000 --regid=1000 --clear-groups cat "$f" >/dev/null 2>&1 && echo "$f"; done | wc -l'
       # EXPECTED: 78   (after remediation: 0, ignoring /nix/store)
 4. **Scrub the copies AND stop regeneration (A-redfix-1, widened).** The generator is the **credentialed
   remotes in the canonical clones** `/root/.abra/recipes/*/.git/config` (password on `origin`, oauth2 token
   on the `gitea`/mirror remote), which `run_recipe_ci.py:348-353`'s `CCCI_SKIP_FETCH=1` `shutil.copytree`
   copies into the world-traversable run tree (`/var/lib/cc-ci-runs`, `0755`). Canonicals are shielded by
   `/root` = `0700`; the run-dir copies are not.
   **REGENERATES WEEKLY (corrected wake #55).** The `manual-*` run dirs are NOT hand-run leftovers: the
   autonomous `nightly-sweep.timer` runs `run_recipe_ci.py` outside Drone (`nightly_sweep.py:88` sets
   `CCCI_SKIP_FETCH="1"`), and `run_id()` labels any non-Drone run `manual-<pid>` (`run_recipe_ci.py:318-319`).
   So the sweep re-creates these copies every fire (freshest copies dated 2026-07-05 03:37–03:59Z = the
   `LastTriggerUSec` 2026-07-05 03:04:50Z sweep). **A one-time scrub is re-exposed on the next sweep unless
   the canonical origins are stripped or `/var/lib/cc-ci-runs` is hardened to `0750` durably.** Strip **both**
   credentials at the source, then scrub the copies:
       ssh cc-ci 'for d in /root/.abra/recipes/*/; do r=$(basename "$d");
         for rem in origin gitea upstream; do
           git -C "$d" remote get-url "$rem" >/dev/null 2>&1 || continue
           git -C "$d" remote set-url "$rem" "$(git -C "$d" remote get-url "$rem" | sed -E "s#://[^@/]+@#://#")"; done; done
       rm -rf /tmp/v
       find /var/lib/cc-ci-runs -path "*/.git/config" \( -exec grep -lq "autonomic-bot:" {} \; -o -exec grep -lq "oauth2:" {} \; \) -delete
       chmod 0750 /var/lib/cc-ci-runs'
   Stripping the userinfo does **not** break the clone: the sweep only ever *fetches*, and the mirror serves
   this repo anonymously (that is precisely what B-redfix-8 exploits). Verified 2026-07-09T09:1xZ —
   `GIT_TERMINAL_PROMPT=0 git -c credential.helper= ls-remote https://git.autonomic.zone/recipe-maintainers/cc-ci.git HEAD`
   → `f0372b8… HEAD`, rc=0, no credential. The sibling `cc-ci-secrets` submodule URL (`.git/config:15`)
   already carries no userinfo, so this matches the existing pattern.
   **Independently cold-verified by the Adversary (REVIEW-redfix.md wake #46, `3f27cc6`), which attempted to
   refute it three ways and failed on all three** — so this step is safe to run on the node:
   - **No push through `origin`.** The only `git push` in the deployed tree (`scripts/recipe-mirror-sync.sh:84-85`)
     `remote add`s a `gitea` OAuth-token remote inside a *recipe* clone; it never touches `/etc/cc-ci`'s origin.
   - **The `secrets` submodule survives the strip.** Its URL already carries no userinfo and
     `cc-ci-secrets.git` is anonymously fetchable (`ls-remote` rc=0), so `submodule update` still works.
   - **Nothing auto-pulls the checkout.** `/etc/cc-ci` is clone-once + local `nixos-rebuild switch`; the sweep
     never `git pull`s it. (`/root/.git-credentials` exists but no `credential.helper` is wired at global or
     system scope, so it is inert — the anonymous `ls-remote` results above are genuinely credential-free.)
 **Rotation blast radius is small — no operational reason to delay.** `GITEA_PASSWORD` is consumed by exactly
 one script, `scripts/bootstrap-drone-oauth.sh` (`git grep -ln GITEA_PASSWORD -- .` returns that script, `docs/
 install.md`, and machine-docs prose; nothing else in the runner). In particular `scripts/recipe-mirror-sync.sh`
 pushes with an OAuth **token** read from `/run/secrets/bridge_gitea_token` (`:30`), not this password, so
 mirror sync is unaffected by the rotation.
 **HOW to confirm BOTH exposures are closed** (each prints a digest; a value other than `3fcea78925015fc9` on
 both, and not the empty-string tell `e3b0c44298fc1c14`, means rotated):
    # exposure 2 — the embedded copy; `python3` is absent on cc-ci, so use sha256sum
    ssh cc-ci 'sed -n "s#^\s*url = https://[^:]*:\([^@]*\)@git\.autonomic\.zone/recipe-maintainers/cc-ci\.git#\1#p" \
      /etc/cc-ci/.git/config | head -1 | tr -d "\n" | sha256sum | cut -c1-16'
 **EXPECTED after step 3:** the command prints **nothing** (no userinfo left to match) — an empty result here
 is the fix, whereas an `e3b0c44298fc1c14` from the *step-2* probe is a broken probe. Do not confuse them.
 **Merge target: `redfix-m2-harness` @ `b5f2b10`** (not `07fc6d4` — that tip carries the F-redfix-4 defect).
 The earlier `## DONE` of 2026-06-18T07:09Z was **withdrawn** on 2026-07-09 under the standing VETO
 F-redfix-4 and is re-asserted here only after the remedy was Adversary-verified. Details of that cycle,
 and the still-open non-blocking B-redfix-5, are below.
 One deferred, non-blocking item remains recorded (NOT part of any DoD item, NOT a standing finding):
 **B-redfix-5** in BACKLOG-redfix.md — in `warm_reconcile.py`, TWO post-`abra.undeploy()` calls sit outside
 the upgrade's `try/except`: `warmsnap.snapshot()` on the upgrade path (`:514`) and `warmsnap.restore()` on
 the rollback path (`:536`). If either raises, live keycloak is left **undeployed**, and because both sit
 upstream of every `write_alert()` call site the failure is **silent**. Pre-dates the enrollment (present at
 `07fc6d4`). Adversary concurred it is not a VETO. Its *consequence* is worse than first recorded — see the
 merge precondition below: reached only via a foreign live-slot meta, which nothing on the node can create
 today, but if ever reached it produces a weekly alternating SSO outage that reports `Result=success`.
 **MERGE PRECONDITION for `b5f2b10`.** The F-redfix-4 fix *adds* a raise path to both of those calls —
 `warmsnap._assert_slot_not_foreign()` (`warmsnap.py:158` in `snapshot()`, `:209` in `restore()`). It raises
 iff the live slot's `snapshot/meta.json` records a `domain` other than the one passed. Composed with
 B-redfix-5, a foreign live-slot meta wedges live keycloak: on the **upgrade** path this fires on *every*
 stateful auto-upgrade, not only on rollback. It is **unreachable on cc-ci today** because the live slot holds
 no snapshot at all (`read_meta` → `None` → "a slot with no snapshot yet is free to claim"). It is unreachable
 for that reason — **not** because F-redfix-4 is closed.
 **THE BINDING RULE: never deploy `07fc6d4`.** (Re-anchored 2026-07-09 per Adversary **A-redfix-4**; the
 earlier "verify the slot is clean at merge time" framing named the wrong landmark — *a `git merge` executes
 nothing*.) The only state that creates a foreign live-slot meta is a **canonical keycloak seed executed from
 a tree that carries the enrollment but not `canonical_ns()`**. Exactly one such tree exists: `07fc6d4`
 (`tests/keycloak/recipe_meta.py` → `WARM_CANONICAL = True`, no `canonical_ns()`). `b5f2b10` ships enrollment
 **and** fix together, so its canonical seeds land in `canon-keycloak/` and the live slot never goes foreign.
 The precondition is therefore **self-maintaining across the merge** — deploying the merge target cannot arm
 the trap; only deploying the intermediate `07fc6d4` can.
 **No autonomous node activity can arm it.** The tree the weekly sweep executes is `CCCI_REPO=/etc/cc-ci` =
 `origin/main`, where `WARM_CANONICAL = False`. Only `07fc6d4` and `b5f2b10` have `True`.
 **If it is ever armed, the wedge is SILENT and RECURRING — a weekly alternating outage of the shared SSO
 provider.** (Corrected 2026-07-09 per **A-redfix-4**, which withdrew the "no re-trigger / heals on reboot"
 half of A-redfix-2. My own earlier "recovery is manual" (`68b51d5`) was also wrong. Re-derived independently
 against the node — see HOW below.) `warm_reconcile.py` has **two** invokers, and the second is a weekly timer:
    nightly-sweep.timer  (OnCalendar=Sun *-*-* 03:00:00, Persistent=true)
      └─ nightly_sweep.main():147 → roll_warm_infra() → subprocess warm_reconcile.py keycloak
         (WARM_APPS = ["keycloak", "traefik"], nightly_sweep.py:39,57-61)
 Once armed, with `current != latest`: **sweep N** → `abra.undeploy()` → `snapshot()` guard raises → keycloak
 **DOWN**. **Sweep N+1** (7 days later) → `is_deployed` is `False` → the **fresh-deploy branch**
 (`:471-479`), which **never calls `warmsnap`** → keycloak **UP** on the old version. **Sweep N+2** → upgrade
 path → **DOWN** again. **Alternating weeks, indefinitely**, because the foreign `snapshot/meta.json` is never
 removed. (`abra.undeploy()` (`abra.py:295-297`) runs only `abra app undeploy -n` and does **not** remove the
 app `.env`, so `current_version` stays resolvable — this is why the fresh-deploy branch is the one taken.)
 **And it raises no alert.** `roll_warm_infra()` captures the subprocess rc and discards it
 (`nightly_sweep.py:59-62` — `rc` is only ever printed), so `nightly-sweep.service` still reports
 `Result=success`. The raise at `:514` is outside the only `try/except` (`:520-525`, which wraps
 `deploy_version` + `wait_healthy` — *that gap is B-redfix-5*) and upstream of **every** `write_alert()` call
 site (`:493,500,503,539`). `reconcile()`'s sole caller `main():556` does not catch either. A weekly SSO
 outage would surface **nowhere**. (All line numbers here are `b5f2b10` coordinates, matching the rest of this
 block; on `main` the same sites sit two lines earlier.)
 **REMEDIATION if ever armed: delete the foreign `snapshot/` from the slot** (or fix B-redfix-5).
 **Redeploying keycloak does NOT fix it** — the next sweep re-wedges. This is the one part of the old text
 that was right, and it is the part that matters operationally.
 **HOW to verify** (from a clone at `b5f2b10`). Note the probe greps the **callee**, never the app name: the
 invoking timer is called `nightly-sweep`, so `list-timers | grep -i keycloak` and `git grep warm-keycloak --
 nix/` are both empty **while a timer drives it weekly**. Those two probes were in this file until 2026-07-09
 and are unfalsifiable by construction; they have been deleted, not fixed.
    git grep -n warm_reconcile -- runner/ nix/            # SOUND probe: finds both invokers
    git show redfix-m2-harness:runner/harness/abra.py     | sed -n '295,297p'   # undeploy: no .env removal
    git show redfix-m2-harness:runner/warm_reconcile.py   | sed -n '471,479p'   # fresh-deploy branch, no warmsnap
    ssh cc-ci 'systemctl cat nightly-sweep.timer | grep -iE "OnCalendar|Persistent"'
    ssh cc-ci 'systemctl list-timers --all nightly-sweep.timer --no-pager'
 **EXPECTED:** `git grep warm_reconcile` prints both `nix/modules/warm-keycloak.nix:26` (the oneshot unit) and
 `runner/nightly_sweep.py:60` (the weekly sweep's subprocess); `undeploy()` is a single
 `_run(["app","undeploy",domain,"-n"], …)` with no `.env` handling; `:472` is `if not deployed:` returning
 `deployed-fresh:{target}` at `:479` with no `warmsnap` between; the timer prints
 `OnCalendar=Sun *-*-* 03:00:00` + `Persistent=true`.
 **Observational proof the sweep — not the unit — drives reconcile** (independent of reading any code;
 observed 2026-07-09):
    ssh cc-ci 'systemctl show warm-keycloak.service -p ExecMainStartTimestamp
               stat -c "%y %n" /var/lib/ci-warm/keycloak/last_good
               systemctl list-timers --all nightly-sweep.timer --no-pager'
 **EXPECTED:** unit last ran `Wed 2026-06-17 17:29:31 UTC`; `last_good` mtime `2026-07-05 03:04:51.209`; timer
 last trigger `Sun 2026-07-05 03:04:50 UTC`. The slot was written **18 days after the unit last ran and 1.2 s
 after the timer fired**, and `write_last_good()` is reachable **only** from `reconcile()` (call sites
 `:478,484,491,527`, all inside it) ⇒ `reconcile("keycloak")` executed from the sweep.
 **Also:** `warm-keycloak.service`'s `ExecStart` resolves to `/nix/store/…-runner/warm_reconcile.py`, and
 `harness/warmsnap.py` lives in that **same** derivation. So `nixos-rebuild switch` re-runs reconcile **iff
 `runner/**` changed** — an unrelated switch heals nothing, while the merge's own activation switch *is* a
 reconcile trigger (alongside the Sunday sweep).
 **Blast radius: keycloak only.** `warm.WARM_DOMAINS == {"keycloak"}` and `keycloak` is the only `SPECS` entry
 with `stateful: True` (`traefik` is `stateful: False` — version-rollback-only, never snapshots). No other
 warm unit reaches either call site.
 **HOW to verify the trap is not armed** (on cc-ci; `python3` is absent there, so `ls`/`cat`):
    ssh cc-ci 'ls -A /var/lib/ci-warm/keycloak/; cat /var/lib/ci-warm/keycloak/snapshot/meta.json 2>&1'
 **EXPECTED (observed 2026-07-09T08:5xZ):** `last_good` only, and `cat` reports `No such file or directory` —
 no `snapshot/`, hence no `meta.json`; no `/var/lib/ci-warm/canon-keycloak` either; live keycloak `200`. **A
 `meta.json` whose `"domain"` is anything other than `warm-keycloak.ci.commoninternet.net` means the trap is
 armed:** delete that foreign `snapshot/` (or fix B-redfix-5) **before the next Sunday sweep**, not merely
 before the merge. A `meta.json` recording `warm-keycloak.ci.commoninternet.net` is self-consistent and passes
 the guard.
 ---
 ### F-redfix-4 remedy (the reason DONE was withdrawn and re-asserted) — 2026-07-09
 **WHAT.** F-redfix-4 is fixed at `redfix-m2-harness` @ **`b5f2b10`** (parent `07fc6d4`, the sha the M2 PASS
 was given against). Warm state is now keyed by a **stack namespace**, not a bare recipe:
 `canonical.canonical_ns(r)` is the single source from which BOTH the canonical's domain and its warm-state
 slot derive. A live-warm provider (`r in warm.WARM_DOMAINS`) gets ns `canon-<r>` → domain
 `warm-canon-keycloak.ci.commoninternet.net` (**unchanged**) + slot `/var/lib/ci-warm/canon-keycloak/`.
 Every other recipe keeps ns `<r>` (the invariant is structural — `r not in warm.WARM_DOMAINS` — not a
 property of any particular count). `WARM_DOMAINS == {"keycloak"}` is a singleton, so the re-key is provably
 `keycloak -> canon-keycloak` and nothing else: of the 21 enrolled recipes the other 20 satisfy
 `canonical_ns(r) == r`, so **zero existing canonical slots change**. On live disk that is 17 seeded
 canonicals (those carrying a `canonical.json`), all of which remain in `prune_stale`'s `keep` set.
 Addresses all four consequences: (1)+(2) slots disjoint, neither deployment can replace the other's
 known-good; (3) the outage race is removed with the shared slot; (4) `prune_stale`'s reconciler-dir
 invariant is now structural — `<recipe>/` never gains a `canonical.json`.
 Also: `warmsnap._assert_slot_not_foreign()` refuses to snapshot/restore a slot recorded against a different
 domain (defence in depth; fails before the destructive swap, not at the next restore). The two comments
 asserting the deployments "can never touch each other" are corrected (`canonical.py`,
 `tests/keycloak/recipe_meta.py`). `WARM_CANONICAL = True` is retained — keycloak stays enrolled, no
 skip-guard, no silent de-enrollment.
 **MIGRATION: none required.** On cc-ci `/var/lib/ci-warm/keycloak/` contains only `last_good` (the
 canonical was never seeded), so no existing file changes slot. Verify: `ls -A /var/lib/ci-warm/keycloak/`
 → `last_good` only.
 **WHERE.** cc-ci repo, branch `redfix-m2-harness` @ `b5f2b10`. Files: `runner/harness/warmsnap.py`,
 `runner/harness/canonical.py`, `runner/warm_reconcile.py`, `runner/run_recipe_ci.py`,
 `tests/keycloak/recipe_meta.py`, `tests/unit/test_warmsnap.py`, `tests/unit/test_canonical.py`.
 **HOW to verify (1) — unit suite, cold clone, no docker needed.**
    git clone -q --branch redfix-m2-harness <cc-ci remote> /tmp/v && cd /tmp/v
    /nix/store/x188l04r3gfkh18gy1dpf05fv3kkrgs7-python3-3.12.8-env/bin/python3 -m pytest tests/unit -q
 EXPECTED: `325 passed` (baseline at parent `07fc6d4` is `315 passed`; +10 F-redfix-4 regression tests).
 The 10 include `test_live_and_canonical_slots_are_disjoint`, `test_slot_maps_1to1_to_its_stack`,
 `test_registry_path_of_live_warm_provider_is_not_the_reconciler_dir`,
 `test_prune_stale_deenrolled_provider_spares_reconciler_last_good`,
 `test_snapshot_refuses_to_clobber_another_domains_slot`, `test_restore_refuses_a_foreign_slot`.
 **HOW to verify (2) — the VETO's clearing condition, on the real node.** Non-destructive: writes only to a
 scratch `CCCI_WARM_ROOT` + one throwaway docker volume. Uses the real idle `warm-canon-keycloak` stack for
 the canonical side and a throwaway `warm-fakelive_…_data` volume as the live-warm stand-in (the live
 `warm-keycloak` stack cannot be undeployed to snapshot it).
    ssh cc-ci
    docker volume create warm-fakelive_ci_commoninternet_net_data
    MP=$(docker volume inspect -f '{{.Mountpoint}}' warm-fakelive_ci_commoninternet_net_data); echo x > $MP/live.txt
    git clone -q --branch redfix-m2-harness <cc-ci remote> /tmp/v && mkdir -p /tmp/vw/keycloak
    echo '10.7.1+26.6.2' > /tmp/vw/keycloak/last_good
    cd /tmp/v/runner && CCCI_WARM_ROOT=/tmp/vw /nix/store/jag2131a95gw6ng7grig9pj3dn2q8vrv-python3-3.12.8-env/bin/python3 - <<'PY'
    import sys; sys.path.insert(0, ".")
    from harness import warmsnap as ws, canonical as c
    LIVE, CANON = ws.live_slot("keycloak"), c.canonical_slot("keycloak")
    CANON_DOM, LIVE_DOM = c.canonical_domain("keycloak"), "warm-fakelive.ci.commoninternet.net"
    print(ws.snap_dir(LIVE), ws.snap_dir(CANON), c.registry_path("keycloak"), sep="\n")
    ws.snapshot(CANON, CANON_DOM, version="canon-known-good")
    ws.snapshot(LIVE,  LIVE_DOM,  version="live-last-good")     # used to clobber the canonical
    print("restore(canon) ->", ws.restore(CANON, CANON_DOM)["volumes"])
    print("restore(live)  ->", ws.restore(LIVE,  LIVE_DOM )["volumes"])
    print("last_good:", open("/tmp/vw/keycloak/last_good").read().strip())
    try: ws.snapshot(CANON, LIVE_DOM); print("FAIL: foreign snapshot allowed")
    except ws.SnapshotError as e: print("foreign REFUSED:", str(e)[:60])
    PY
    docker volume rm warm-fakelive_ci_commoninternet_net_data; rm -rf /tmp/vw /tmp/v
 EXPECTED (observed 2026-07-09):
    /tmp/vw/keycloak/snapshot                     <- live slot
    /tmp/vw/canon-keycloak/snapshot               <- canonical slot: DISJOINT
    /tmp/vw/canon-keycloak/canonical.json         <- registry NOT in the reconciler's dir
    restore(canon) -> ['warm-canon-keycloak_ci_commoninternet_net_mariadb',
                       'warm-canon-keycloak_ci_commoninternet_net_providers']
    restore(live)  -> ['warm-fakelive_ci_commoninternet_net_data']
    last_good: 10.7.1+26.6.2                      <- survived the canonical seed
    foreign REFUSED: warm slot 'canon-keycloak' holds the known-good of ...
 Each `restore()` returns its OWN stack's volumes (the clearing condition). Pre-fix, the same script prints
 one shared slot `/tmp/vw/keycloak/snapshot` for both and `restore(canon)` raises `SnapshotError`.
 **Node integrity after my own run of the above (2026-07-09).** Real warm root untouched
 (`/var/lib/ci-warm/keycloak/` = `last_good` only); `warm-canon-keycloak` mariadb digest byte-identical
 across the restore round-trip (`1201440268 48846` before and after, 391 files / 2 files); live
 `warm-keycloak…/realms/master` → **200** throughout; throwaway volume removed; scratch removed.
 **Scope.** keycloak only. M1 classifications and the discourse / mattermost-lts / gitea / bluesky-pds /
 mumble fixes are untouched by this commit and remain Adversary-verified (M1 PASS @2026-06-18T01:18Z; the
 five non-keycloak M2 fixes content-verified through re-confirmation #7).
 **Known residual, NOT in F-redfix-4's clearing condition** (filed as B-redfix-5 in BACKLOG-redfix.md, not
 blocking): `warm_reconcile.py`'s rollback `warmsnap.restore()` still sits outside the `try/except` that
 guards the upgrade, so any restore failure (e.g. a corrupt/absent snapshot) leaves live keycloak undeployed
 after `abra.undeploy()`. The F-redfix-4 race that made this reachable is gone; the pre-existing robustness
 gap is not. Recorded, not silently dropped.
 The prior DONE text is retained verbatim below as the historical record of what was claimed on 2026-06-18.
 ---
 ## (HISTORICAL — withdrawn 2026-07-09 under VETO F-redfix-4; superseded by the DONE above) DONE — 2026-06-18T07:09Z
 Phase `redfix` COMPLETE. All six canon-sweep failures investigated in isolation, root-caused,
 classified, **FIXED — each via a recipe PR or a harness improvement — and verified green**; no recipe
@ -475,29 +26,6 @@ M1 read-only crash gone; bluesky-pds recipe PR #4 @4987ba9 (caddy `${STACK_NAME}
 200 (was 000). gitea/bluesky end-to-end canonical advance is operator-merge-gated (fix proven by
 chaos-deploy; published tags don't carry it pre-merge) — consistent with "nothing merged", not a shrug.
 ### Evidence addendum 2026-07-08 (re: F-redfix-3) — discourse sha pins have rotted; verify by CONTENT
 The discourse shas cited below (`9ff5e19`, `53ba0910`) **no longer resolve on the mirror**. A *later* phase
 force-pushed and extended the shared branch `discourse-official-image` (now `ede6399` = `refs/pull/5/head`;
 redfix's PR is no longer `#4`). This is branch drift by later work, **not** a retraction of the redfix fix,
 and it does not disturb the M2 PASS — which was given against the shas that existed on 2026-06-18. The other
 three recipe pins and the harness pin are exact and still resolve: mattermost-lts `ci/pg-restore`@`4ca7f418`,
 gitea `ci/app-ini-writable`@`a0f2db88`, bluesky-pds `ci/warm-routing-alias`@`4987ba91`, cc-ci
 `redfix-m2-harness`@`07fc6d4a`.
 Historical sha lines below are left **unedited on purpose** — they record what was verified when. Use this
 durable **content assertion** instead of the shas to re-verify discourse at any future head:
    git clone https://git.autonomic.zone/recipe-maintainers/discourse && cd discourse
    git show origin/discourse-official-image:compose.yml | grep -m1 'image:.*discourse'
    git show origin/discourse-official-image:compose.smtpauth.yml | grep -c sidekiq
 EXPECTED: `image: discourse/discourse:3.5.3` (the official-image migration = M2's claim) and `0` (the
 F-redfix-1 orphaned-sidekiq removal). Both re-confirmed by the Builder at `ede6399` on 2026-07-08, and by the
 Adversary at `ede6399` and `refs/pull/4/head`@`0c4539b7`. (Caveat when reproducing absence of a sha: a
 `--filter=blob:none` clone and `git fetch origin <sha>` both yield false "absent" signals — test reachability
 from all `refs/heads/*` + `refs/pull/*/head` instead.)
 ---
 ## Phase: M1 — investigate + isolate + classify (IN PROGRESS)
@ -707,33 +235,6 @@ in the two sections above ("M1 results table" and "HOW the Adversary cold-verifi
 Evidence logs on cc-ci: `/tmp/redfix-{discourse,mattermost-lts,mumble,mumble2,bluesky-pds,gitea2}.log`.
 Reasoning/dead-ends in JOURNAL-redfix.md. Node left clean (see "Node state left clean" above).
 ## Untracked file `main.go` — NOT produced by this phase (operator: unexplained)
 **WHAT.** An untracked `main.go` (281 bytes, a Go "Hello, World!" `net/http` server on `:8080`) sits in the
 repo root. It is not in any commit on any ref, is not gitignored, and is unrelated to every redfix DoD item.
 No `go.mod` accompanies it. I did not create it and have not committed, modified, or deleted it.
 **WHERE.** `/srv/cc-ci-orch/cc-ci/main.go` — **one file in one clone.**
 **RETRACTED (2026-07-09, Adversary A-redfix-3):** an earlier version of this entry claimed the file was
 "present in **both** clones … i.e. written by something outside git." **That evidence is void.** `/srv/cc-ci`
 is a **symlink to `/srv/cc-ci-orch`**, so `/srv/cc-ci/cc-ci/main.go` and `/srv/cc-ci-orch/cc-ci/main.go` are
 the **same inode** — one file seen twice, not two clones. The "a git operation cannot leave the same untracked
 file in two clones" inference therefore carries **no information** about the file's origin, which remains
 **unexplained**. Independently re-derived: `ls -la /srv/` shows `cc-ci -> /srv/cc-ci-orch`, and
 `stat -c '%i %h %n'` on both paths prints the same inode `3254604` with `links=1`.
 **HOW to verify.** `git log --all --oneline -- main.go` → empty (in no commit).
 `git status --short` → `?? main.go`.
 `ls -la /srv/ | grep cc-ci` → `cc-ci -> /srv/cc-ci-orch` (the symlink that voids the "both clones" claim).
 `stat -c '%i %h %n' /srv/cc-ci/cc-ci/main.go /srv/cc-ci-orch/cc-ci/main.go` → same inode, `links=1`.
 **Risk: inert** (Adversary-confirmed). 281-byte hello-world `net/http` listener; no `go.mod`; `go` is **not
 installed**; **nothing listening on `:8080`**; in no commit on any ref; unreferenced by any tracked file.
 **EXPECTED.** Left in place, untracked. Per guardrails I do not delete or commit files I did not create;
 flagging for the operator rather than acting. It affects no gate, no DoD item, and no verdict.
 ## Blocked
 (none)
--- a/scripts/recipe-mirror-sync.sh
+++ b/scripts/recipe-mirror-sync.sh
@ -36,10 +36,7 @@ RECIPE_DIR="${HOME}/.abra/recipes/${RECIPE}"
 }
 TOKEN="$(tr -d '[:space:]' <"${TOKEN_FILE}")"
 API="https://${GITEA_HOST}/api/v1"
-# Push over SSH (git@:2222, key in ~/.ssh) so no credential is ever written into the recipe
+MIRROR_PUSH="https://oauth2:${TOKEN}@${GITEA_HOST}/${NAMESPACE}/${RECIPE}.git"
 # clone's .git/config — that config gets copytree'd into the world-readable per-run tree
 # (CCCI_SKIP_FETCH staging), which previously leaked the token. API still uses the token below.
 MIRROR_PUSH="git@${GITEA_HOST}:${NAMESPACE}/${RECIPE}.git"
 auth=(-H "Authorization: token ${TOKEN}")
 # Ensure the recipe is cloned (abra recipe fetch clones from the catalogue → upstream).
--- a/2
+++ b/2
--- a/tests/libredesk/custom/test_health.py
+++ b/tests/libredesk/custom/test_health.py
@ -0,0 +1,20 @@
 """libredesk — health check: the app's /health endpoint responds 200 through Traefik.
 This is the same endpoint the compose healthcheck hits internally (:9000/health); exercising it via
 the public domain confirms the app is bound and the proxy route is wired.
 """
 from __future__ import annotations
 import os
 import sys
 sys.path.insert(0, os.path.join(os.path.dirname(__file__), "..", "..", "..", "runner"))
 from harness import http as harness_http  # noqa: E402
 def test_libredesk_health(live_app):
    """GET /health → 200."""
    url = f"https://{live_app}/health"
    status, _ = harness_http.retry_http_get(url, expect_status=(200,), max_wait=120, interval=5)
    assert status == 200, f"GET {url} HTTP {status} (expected 200)"
--- a/tests/libredesk/custom/test_ui.py
+++ b/tests/libredesk/custom/test_ui.py
@ -0,0 +1,49 @@
 """libredesk — UI probe: the served root HTML is the real LibreDesk app, not a fallback page.
 /health passing only proves the process is up. This asserts the front-end SPA is actually bound and
 serving its own shell — a wedged backend or a misrouted proxy would 200 with generic/empty content.
 """
 from __future__ import annotations
 import os
 import ssl
 import sys
 import urllib.request
 sys.path.insert(0, os.path.join(os.path.dirname(__file__), "..", "..", "..", "runner"))
 from harness import http as harness_http  # noqa: E402
 _CTX = ssl.create_default_context()
 _CTX.check_hostname = False
 _CTX.verify_mode = ssl.CERT_NONE
 def _get_body(url: str) -> tuple[int, str]:
    req = urllib.request.Request(url, method="GET")
    with urllib.request.urlopen(req, timeout=15, context=_CTX) as r:
        return r.status, r.read().decode(errors="replace")
 def test_libredesk_serves_app_shell(live_app):
    """GET /; assert LibreDesk-specific brand or SPA-asset markers in the served HTML."""
    url = f"https://{live_app}/"
    def _ready():
        try:
            status, body = _get_body(url)
        except Exception:  # noqa: BLE001
            return None
        return body if status in (200, 301, 302) else None
    body = harness_http.assert_converges(_ready, f"GET {url}", max_wait=120, interval=5)
    lower = body.lower()
    # Brand markers: the app title / meta / bundle references carry "libredesk".
    brand = [m for m in ("libredesk", "libre desk") if m in lower]
    # SPA asset markers: the built front-end serves hashed JS/CSS bundles + a favicon.
    assets = [m for m in ("/assets/", "favicon", ".js", ".css", "<div id=\"app\"", "id=app") if m in body]
    assert brand or assets, (
        f"GET {url} HTML has no LibreDesk brand or SPA-asset markers. Excerpt: {body[:300]!r}"
    )
--- a/tests/libredesk/install_steps.sh
+++ b/tests/libredesk/install_steps.sh
@ -0,0 +1,49 @@
 #!/usr/bin/env bash
 # libredesk — INSTALL-TIME secret hook.
 #
 # Runs during the install tier AFTER `abra app new` + EXTRA_ENV + `abra app secret generate --all`
 # and BEFORE the single `abra app deploy` (lifecycle.py::_run_install_steps). LibreDesk validates
 # two secrets with formats that abra's generic generator does not guarantee, so we insert compliant
 # values here (at a bumped version — swarm forbids overwriting a secret at the same version) and
 # point the .env at them, so the recipe deploys ONCE, healthy, with no reconverge:
 #   - enc_key   : exactly 32 hex chars (AES-256 encryption_key)         → openssl rand -hex 16
 #   - admin_pwd : 10–72 chars incl. upper+lower+number+special (the app rejects weaker "System"
 #                 user passwords, which abra's alphanumeric generator can produce)
 # db_password is fine as abra-generated (no special format), so it is left untouched.
 #
 # Env supplied by the harness:
 #   CCCI_APP_DOMAIN — the per-run libredesk app domain (== abra app name)
 #   CCCI_APP_ENV    — path to the app's .env (the one `abra app deploy` reads)
 set -euo pipefail
 : "${CCCI_APP_DOMAIN:?missing}"
 ENV_PATH="${CCCI_APP_ENV:?missing}"
 # Insert a compliant value for <secret> and repoint SECRET_<VAR>_VERSION at the bumped version.
 insert_secret() {
  local secret="$1" env_var="$2" value="$3"
  local cur new num
  cur=$(grep -E "^\s*${env_var}=" "$ENV_PATH" | tail -1 | cut -d= -f2 | tr -d '"\r' || echo "v1")
  cur=${cur:-v1}
  num=$(( ${cur#v} + 1 )); new="v${num}"
  # abra forbids overwriting an existing version; insert at the fresh version. -C creates it, -o
  # allows overwrite if a stale one exists, --no-input for non-interactive.
  local log
  log=$(abra app secret insert "$CCCI_APP_DOMAIN" "$secret" "$new" "$value" --no-input -C -o 2>&1) ||
    log=$(script -qec "abra app secret insert $CCCI_APP_DOMAIN $secret $new $value --no-input -C -o" /dev/null 2>&1) ||
    { echo "  install_steps: abra app secret insert ${secret}@${new} failed: $log"; exit 1; }
  sed -i "s|^\s*${env_var}=.*|${env_var}=${new}|" "$ENV_PATH"
  echo "  install_steps: ${secret} inserted at ${new} (was ${cur})"
 }
 # 32 hex chars = 16 bytes (AES-256 key LibreDesk expects for encryption_key).
 ENC_KEY=$(openssl rand -hex 16)
 # 10–72 chars with upper+lower+number+special. Build deterministically-compliant: a fixed
 # upper+lower+digit+special prefix plus random hex entropy (lowercase letters + digits).
 ADMIN_PWD="Ci$(openssl rand -hex 12)Aa1!"
 insert_secret enc_key   SECRET_ENC_KEY_VERSION   "$ENC_KEY"
 insert_secret admin_pwd SECRET_ADMIN_PWD_VERSION "$ADMIN_PWD"
 echo "  libredesk install_steps: enc_key (32-hex) + admin_pwd (complex) inserted; deploy will use them"
--- a/tests/libredesk/recipe_meta.py
+++ b/tests/libredesk/recipe_meta.py
@ -0,0 +1,32 @@
 # Per-recipe harness config for libredesk (LibreDesk — open-source, self-hosted customer-support /
 # help desk; recipe repo recipe-maintainers/Libre-Desk, abra TYPE=libredesk).
 #
 # Stack (compose.yml): app (libredesk/libredesk:v2.4.0, serves on :9000 behind Traefik) + db
 # (postgres:17-alpine, user/db=libredesk) + redis (redis:7-alpine). abra-entrypoint.sh self-installs
 # the schema and runs migrations on boot (`--install --idempotent-install` then `--upgrade`), so the
 # stack comes up ready with no post-deploy step.
 #
 # Secrets have STRICT formats that abra's generic generator does not satisfy, so install_steps.sh
 # inserts compliant values before the single deploy (see that file):
 #   - enc_key   must be 32 hex chars (AES-256 key)         → openssl rand -hex 16
 #   - admin_pwd must be 10–72 chars, upper+lower+num+sym   → the "System" user's password
 #   - db_password is abra-generated (no special format) and left as-is.
 HEALTH_PATH = "/health"        # the app's own healthcheck endpoint (compose hits :9000/health)
 HEALTH_OK = (200,)
 DEPLOY_TIMEOUT = 600           # app + postgres + redis; app runs install+migrate on first boot
 HTTP_TIMEOUT = 300
 BACKUP_CAPABLE = True          # db carries backupbot.backup labels (pg_dump pre-hook / psql restore-hook)
 # Only one published version exists so far (0.1.0+v2.4.0), so there is no PRIOR deployable base to
 # upgrade FROM — the upgrade rung is intentionally N/A (a declared skip, never promoted to a pass).
 # Drop this once a second version is published; the dynamic base then resolves it (last-green warm
 # canonical → same-version step-back → main tip).
 EXPECTED_NA = {
    "upgrade": "only one published version so far (0.1.0+v2.4.0) — no prior deployable base to "
    "upgrade FROM yet; drop this when a second version ships",
 }
 # canon §2.B: enroll as a DATA-WARM canonical (all recipes enrolled — operator 2026-06-17).
 # The weekly sweep promotes this recipe's canonical to its latest green RELEASE TAG.
 WARM_CANONICAL = True