cc-ci/machine-docs/BACKLOG-redfix.md

BACKLOG — phase redfix

Build backlog

M1 — investigate + isolate + classify (all six)

• discourse — reproduce cold-deploy timeout/wedge in isolation; root-cause (headroom vs convergence bug vs upstream compose defect sidekiq.depends_on: discourse); classify.
• mattermost-lts — test_restore.py::test_restore_returns_state in isolation: green→load flake, red→diagnose restore (recipe vs test).
• mumble — custom/test_protocol_handshake.py::test_handshake_completes_with_channel_presence in isolation (canonical already present from today → likely flake; confirm).
• bluesky-pds — warm-canonical promote routing: why warm-bluesky-pds… → 000 over HTTPS while container healthy internally + cold-test domain routes. Find cc-ci warm-machinery defect.
• gitea — 3.5.3→3.6.0 warm advance crash (app.ini read-only, JWT save). Recipe vs harness.
• keycloak — de-enrolled (live-warm OIDC collision). Design collision-free warm domain/namespace.

M2 — FIX + verify all six (recipe PR or harness improvement)

Execution gated on M1 PASS (avoid node contention with Adversary M1 re-runs; classifications must hold). Concrete fix designs from M1 evidence:

• mattermost-lts (recipe PR, clearest) — add pg_backup.sh (immich pattern, no VectorChord bits): backup(){ pg_dump -U mattermost mattermost | gzip > /var/lib/postgresql/data/backup.sql; } restore(){ gunzip -c …/backup.sql | psql -U mattermost -d mattermost -f -; }. compose: add configs: pg_backup → /pg_backup.sh; postgres labels → backup.pre-hook: /pg_backup.sh backup, restore.post-hook: /pg_backup.sh restore, backup.volumes.postgres.path: backup.sql (dump-only, drop the whole-PGDATA backup.path + the rm post-hook). Verify via !testme → restore green.
• bluesky-pds (recipe PR) — eliminate the app-alias collision on shared proxy: give the PDS service a unique name (e.g. pds) OR a unique network alias, and update caddy refs (reverse_proxy, on_demand_tls ask http://…/tls-check), healthcheck, backup labels, ops/test service= refs. Verify warm promote → 200 on /xrpc/_health. (NOTE: cc-ci harness ops.py/tests reference service="app" for bluesky? check + update if the recipe service renames — but recipe mirror is PR-only; cc-ci-side refs are a separate cc-ci change.) Confirm exact approach in M2.
• gitea (recipe PR) — make app.ini writable on the warm-reattach advance so 3.6.0 can persist the JWT secret: render app.ini into the WRITABLE config:/etc/gitea volume via the existing docker-setup.sh entrypoint (copy the templated config to a writable path) instead of the read-only app_ini docker-config mount; OR ensure the persisted JWT secret is accepted without rewrite. Verify the 3.5.3→3.6.0 advance promotes. (Ties to LFS PR #1.)
• keycloak (harness, cc-ci branch) — canonical.canonical_domain(r): return a collision-free domain when r is a live-warm provider (r in warm.WARM_DOMAINS) → e.g. warm-canon-<r>.ci.commoninternet.net; else keep warm-<r> (zero blast radius on the 15 others). Set keycloak WARM_CANONICAL=True. Verify keycloak promotes at warm-canon-keycloak WITHOUT disrupting live warm-keycloak (200 throughout).
• mumble (harness, cc-ci branch) — stabilize the handshake under load: add a READY_PROBE/ readiness gate (TCP 64738 stably listening + a successful handshake) before the custom tier and/or raise retry_handshake budget; verify green under a concurrent-load re-run.
• discourse (TRICKIEST — decide in M2) — the overlay test_upgrade.py asserts a bitnamilegacy→official migration absent from all releases/main. Options: (a) cc-ci test PR (--with-tests) scoping the faithfulness assertion to ONLY fire when the head actually performs the migration (image still bitnamilegacy → N/A, not RED) — NOT a weakening, a correct scope; + file an upstream recipe issue/PR for the real bitnamilegacy→official migration. (b) recipe PR doing the migration (major rewrite — official discourse image is launcher-based, likely infeasible cleanly). Lean (a)+tracked-upstream; may need operator input (DEFERRED?) — assess in M2.

M3 — post-VETO remediation (F-redfix-4)

• keycloak warm-state slot collision — FIXED at redfix-m2-harness@b5f2b10. canonical_ns() is now the one namespace behind both the canonical's domain and its warm-state slot; live-warm provider → canon-<recipe> slot, disjoint from the reconciler's <recipe>/. Plus a naming-independent _assert_slot_not_foreign() guard. Unit suite 315→325; clearing condition re-run green on cc-ci (each restore() returns its own stack's volumes; reconciler last_good survives). Verify per STATUS-redfix.md "Gate: M2 RE-CLAIMED".

• B-redfix-5 — reconciler's post-undeploy warmsnap calls are outside the upgrade's try/except (NOT blocking; NOT part of F-redfix-4's clearing condition; recorded so it is not silently dropped). In warm_reconcile.py there are two such sites, both after abra.undeploy(domain) and both outside the try/except that guards the upgrade: (a) upgrade path :512-514 — abra.undeploy → wait_undeployed → warmsnap.snapshot(...); (b) rollback path :534-536 — abra.undeploy → wait_undeployed → warmsnap.restore(...) → deploy_version(last_good) at :537. If either raises for ANY reason (absent/corrupt snapshot, foreign slot, docker error, "no volumes found") the exception propagates, the following deploy_version never runs, and live keycloak is left undeployed. Site (a) is on the normal upgrade path — it does not need a rollback to fire. Reachability, corrected (wake #42): F-redfix-4's fix does not remove the trigger, it adds one — _assert_slot_not_foreign() is a new raise inside BOTH snapshot() (warmsnap.py:158) and restore() (:209). It is unreachable on cc-ci today because the live slot holds no snapshot/meta.json (read_meta → None, and an unclaimed slot is free to claim) — not because F-redfix-4 is closed. A pre-fix (07fc6d4) canonical seed is the one state that writes a foreign domain into the live slot; see the MERGE PRECONDITION in STATUS-redfix.md. Consequence, corrected (wake #44, A-redfix-4) — worse than recorded above, still not blocking. The wedge is neither manual-recovery-only (my 68b51d5) nor reboot-healed (A-redfix-2). A weekly nightly-sweep.timer (OnCalendar=Sun *-*-* 03:00:00, Persistent=true) re-invokes warm_reconcile.py keycloak via nightly_sweep.roll_warm_infra() (:60), so once armed the wedge alternates DOWN/UP every 7 days, indefinitely: sweep N wedges at site (a); sweep N+1 takes the fresh-deploy branch (:471-479, no warmsnap) and comes back up on the old version; sweep N+2 wedges again. It is also silent — roll_warm_infra() discards the subprocess rc (:59-62) so nightly-sweep.service reports Result=success, and site (a) is upstream of every write_alert() (:493,500,503,539). A weekly outage of the shared SSO provider would surface nowhere. Verified first-hand against the node, not adopted from the finding. Not armed, and not self-arming: origin/main — the tree the sweep executes (CCCI_REPO=/etc/cc-ci) — has WARM_CANONICAL = False; only 07fc6d4 and b5f2b10 have True, and b5f2b10 ships the canonical_ns() fix so its seeds land in canon-keycloak/. The binding rule is never deploy 07fc6d4. Remedy sketch: wrap both sites so a warmsnap failure still redeploys last_good (or, if restoring data is judged mandatory before redeploy, alert loudly + leave a breadcrumb rather than dying mid- rollback), and make roll_warm_infra() propagate a non-zero rc so the failure is at least visible. Needs a decision on which is safer for a DB-backed app after a forward migration — that trade-off is why this is filed, not fixed inline. Fixing it now would move the merge target off b5f2b10, against which the M2 PASS was given, so it stays deferred for the operator.

• B-redfix-6 — canonical_ns() docstring says "the 15 existing canonicals"; the real number is 17, and the invariant is not a count (COSMETIC; docs-only; no behaviour change). At redfix-m2-harness@b5f2b10, runner/harness/canonical.py:52 reads "zero blast radius on the 15 existing canonicals". Verified on live disk: /var/lib/ci-warm/ has 20 slots, 17 carrying a canonical.json (spared: alerts, keycloak, traefik — the reconciler dirs). More importantly the guarantee is structural, not numeric: WARM_DOMAINS is a singleton ({"keycloak"}), 21 recipes are enrolled, so exactly one re-keys (keycloak -> canon-keycloak) and the other 20 satisfy canonical_ns(r) == r. The docstring's count will rot again on the next enrollment. Deliberately NOT fixed inline: amending it would move the branch tip off b5f2b10, the exact sha the M2 PASS was granted against and that every drift sweep pins. Fold into the next commit that moves the branch for a substantive reason (e.g. B-redfix-5), rewording to the structural form. Corrected in STATUS-redfix.md prose (which is not sha-pinned) as of wake #17.

• B-redfix-7 — orphaned non-Nix clone at /etc/cc-ci on the node stores the Gitea bot password in plaintext in a world-readable .git/config (OPERATOR CALL; out of redfix scope; found wake #18 while checking a claim in re-confirmation #16). Facts, all read-only-verified on cc-ci: - /etc/cc-ci/.git/config is mode 644 root:root and its origin URL embeds https://autonomic-bot:<password>@git.autonomic.zone/... — the bot's Gitea credential at rest in cleartext. (Value deliberately not reproduced here.) - Severity is LOW, not nil — but my first rationale for that was WRONG (corrected wake #19 after Adversary re-confirmation #17; filed Adversary-side as A-redfix-1). I originally argued "no non-root login users, so only root can read it." That is the wrong axis: a process needs no login shell to run as uid 1000, and mode 644 permits any uid to read. Verified directly — setpriv --reuid=1000 … /etc/cc-ci/.git/config succeeds in the host namespace. - What actually contains the blast radius is mount-namespace isolation, re-derived first-hand: exactly ONE non-root process shares pid1's mount ns (dbus-daemon, uid 4). Every other non-root process is containerized — notably the uid-1000 Quarkus java that IS the internet-facing warm-keycloak: its ns/mnt is 4026533305 vs pid1's 4026531841, and ls /proc/<pid>/root/etc/cc-ci → No such file or directory. Mode-permits ≠ reachable. - Therefore the containment is load-bearing on a fragile property and dies silently if a non-root host-namespace daemon ever appears, or if /etc is bind-mounted into any container. That fragility — not the current reachability — is the reason to fix it. - /etc/cc-ci is a real directory, not a /nix/store symlink, and no systemd unit references it — i.e. undeclared, unmanaged server state that nothing runs from. It violates the standing "keep server state Nix-declared and reversible" rule. NOT actioned by me, deliberately. The Gitea bot credential is a Class-A1 EXTERNAL infra input (plan §4.4) — not mine to rotate or invent. And I did not create /etc/cc-ci, so it is not mine to delete. Remedies for the operator, cheapest first: (a) chmod 600 the config; (b) move the credential to a credential.helper / netrc outside the repo; (c) if the clone is genuinely orphaned, remove it and let Nix own any needed checkout. Rotating the bot password is worth considering regardless, since it has sat in cleartext on disk.

• B-redfix-8 — the live Gitea bot password was committed to this repo, pushed to origin/main, and is served to the UNAUTHENTICATED PUBLIC INTERNET (SEVERITY HIGH — URGENT operator rotation, not deferrable; found + redacted wake #19; public-exposure confirmed wake #20). Public exposure — independently verified wake #20 (not taken from the Adversary's report): a plain urllib.request.urlopen (no auth handler, no netrc, no git credential helper) of …/recipe-maintainers/cc-ci/raw/commit/14c7dee/machine-docs/BACKLOG-redfix.md returns HTTP 200, 33408 bytes, with the cleartext password in the body (sanity-checked: body starts # BACKLOG, contains the A-redfix-1 Repro block — a real fetch, not an error page). Cross-check that the fetch is intact: git cat-file -s 14c7dee:machine-docs/BACKLOG-redfix.md → 33408; a raw blob at a fixed commit sha is immutable, so the served size MUST equal the object size. If your probe returns 33408, that is the leak, not an error page. (Both figures were recorded as 33080 until 2026-07-09T02:24Z — a transcription slip caught by the Adversary and re-measured against the blob; see REVIEW re-confirmation #22.) The mirror is public. So the leaked credential is now world-readable to anyone on the internet, with no account, permanent in history, replicated to every clone, AND push-capable to recipe-maintainers/*. STILL THE LIVE CREDENTIAL — measured, not assumed (wake #23, 2026-07-09). Public reachability (HTTP 200) and the value being unrotated are two different claims; until wake #23 only the first had ever been measured, and "unrotated" was carried forward by repetition. Direct check: the current GITEA_PASSWORD from /srv/cc-ci/.testenv is present verbatim in the 14c7dee blob → True. So the bytes the public internet serves ARE the password the harness authenticates with today; the exposure is not inert. Operator re-check without printing the secret (verified to run, wake #23 — copy verbatim):

      python3 -c 'import hashlib,re;v=re.search(r"^GITEA_PASSWORD=(.*)$",open("/srv/cc-ci/.testenv").read(),re.M).group(1).strip().strip(chr(34)).strip(chr(39));print(hashlib.sha256(v.encode()).hexdigest()[:16])'
  
  → prints `3fcea78925015fc9` **while still unrotated**. A different digest ⇒ rotation happened ⇒
  `14c7dee` is inert ⇒ this item can be closed. (The digest commits to the leaked value without
  republishing it; `sha256` of the raw password, first 16 hex chars.) HEAD (`main`) and the redaction
  commit `e99e2b3` were re-fetched publicly and are **clean** — only the historical commit `14c7dee`
  serves it. This lifts A-redfix-1/B-redfix-8 from LOW (host-local, mount-ns-contained) to **HIGH**:
  mount-namespace isolation is irrelevant once the same secret is on the open web.
  While documenting A-redfix-1, `machine-docs/BACKLOG-redfix.md` gained a "repro" line that inlined the
  **actual bot password** as a `grep` pattern. Introduced by commit **`14c7dee`**, which is **on
  `origin/main`** — i.e. pushed to `git.autonomic.zone/recipe-maintainers/cc-ci`.
  **This strictly escalates A-redfix-1.** That finding was a secret in one 0644 file on one host,
  reachable only from pid1's mount namespace. This moved the same secret into a **git repository**, which
  is: replicated to every clone (both loops' clones and the node's `/etc/cc-ci`), readable by anyone with
  read access to the mirror, and — the sharp edge — the credential grants **push** access to
  `recipe-maintainers/*`, so it now sits inside a repo it can write to.
  **Done:** redacted from `HEAD` (wake #19); the repro now greps `autonomic-bot:` instead, verified to
  return the same `1`, so the finding lost no verifiability.
  **NOT done, and cannot be by me:** the secret remains in **history at `14c7dee` forever**. Excising it
  needs a history rewrite + `--force`, which the standing rules forbid ("Never `--force`"), and which
  would break both loops' clones. Redaction at HEAD stops propagation; it does **not** undo disclosure.
  **⇒ Rotate the `autonomic-bot` Gitea password — URGENT.** It is a live, world-readable, push-capable
  credential; every hour it stays valid it can be used by anyone who has fetched that public URL. It is a
  Class-A1 external input, so only the operator can rotate it. **Rotation is the ONLY remediation that
  actually closes this** — a history rewrite is both forbidden here (`--force`) and insufficient anyway,
  because a value already served publicly must be presumed captured/cached/crawled and cannot be
  un-published. Once rotated, `14c7dee` becomes inert. The re-issued credential must NOT go back into a
  remote URL — see A-redfix-1's remedy ladder (`credential.helper` / netrc outside the repo).
  **Process lesson (mine):** my pre-commit guard did catch this, but I had chained it with `;` instead of
  `&&`, so the commit proceeded anyway and I pushed on top of the leak. A guard whose failure does not
  halt the pipeline is decoration. Fixed by making the check `&&`-gated before `git commit` in wake #19+.
  

• B-redfix-9 — harden CCCI_SKIP_FETCH staging: it copies credentialed .git/config into a world-readable run tree. (DEFERRED — post-merge; outside redfix DoD; no gate impact. Filed wake #54.) fetch_recipe's staging branch (runner/run_recipe_ci.py:348-353) does shutil.copytree(~/.abra/recipes/<recipe>, <run-dir>). The canonical clones carry the bot password AND a live oauth2 token in their remotes but are shielded by /root = 0700; the run tree /var/lib/cc-ci-runs is 0755, so the copy loses the protection of its source — 133 world-readable cred-bearing copies today (78 password, 117 token, overlapping). CORRECTED (wake #55): this regenerates WEEKLY, not just on hand-runs. The manual-* prefix does not mean hand-run: the autonomous nightly-sweep.timer runs run_recipe_ci.py outside Drone (nightly_sweep.py:88 sets CCCI_SKIP_FETCH="1"), and run_id() labels every non-Drone run manual-<pid> (run_recipe_ci.py:318-319). So the sweep itself is the generator and re-arms the exposure each fire. (My earlier "0 production runs carry it, all manual-* = leftovers" mis-read manual-*; the normal Drone fetch path IS clean — clean URL + per-command http.extraHeader token, landed 9b33fdf — but the sweep does not take that path.) Fix: after copytree, strip userinfo from the copied remote (git -C <dest> remote set-url origin <clean>), or exclude .git/config from the copy and re-init the remote clean; plus chmod 0750 /var/lib/cc-ci-runs. Do not rely on rotating the password: rotation without this re-exposes the new value on the next staged run.

Adversary findings

(Adversary-owned — do not edit.)

A-redfix-1 [adversary] — /etc/cc-ci/.git/config is 0644 and embeds the Gitea bot password in cleartext (severity LOW, but for a different reason than B-redfix-7 states)

Independently confirmed the Builder's B-redfix-7. Out of redfix scope; no VETO, does not reopen the phase (pre-existing undeclared server state, not created by redfix, not covered by any DoD item). Filed so the severity rationale is right, because the wrong rationale would let the mitigation evaporate silently.

Repro (read-only, from any shell with ssh cc-ci):

stat -c '%a %U:%G %n' /etc/cc-ci/.git/config      # -> 644 root:root
grep -c 'autonomic-bot:' /etc/cc-ci/.git/config    # -> 1   (cleartext bot password in origin URL)
# [REDACTED by Builder, wake #19] the line above originally inlined the LIVE bot password as the
# grep pattern. Matching on the username is an equivalent repro and leaks nothing. See B-redfix-8.
stat -c '%a %n' / /etc /etc/cc-ci /etc/cc-ci/.git  # -> 755 on every parent: world-traversable

Why LOW — the correct reason. B-redfix-7 argues "no non-root login users exist (no uid>=1000 with a real shell)". That is the wrong axis: the risk is any non-root code execution, login shell or not. The mode genuinely permits it —

setpriv --reuid=1000 --regid=1000 --clear-groups cat /etc/cc-ci/.git/config   # -> prints the password

— so the file is readable by uid 1000. What actually holds the severity down is mount-namespace isolation, which neither of us had checked: every non-root process on the box is containerized and its /etc is not the host's. I enumerated it directly (compare /proc/<pid>/ns/mnt against /proc/1/ns/mnt): of all non-root processes, exactly one lives in the host mount namespace — dbus-daemon, uid 4 (messagebus), not network-facing. Everything else (incl. the uid-1000 java = the warm-keycloak Quarkus container) is in its own namespace and cannot reach host /etc/cc-ci; verified: ls /proc/<keycloak-pid>/root/etc/cc-ci/.git/config -> No such file or directory.

Correction to my own probe. My first pass read the uid-1000 java process off ps, saw setpriv as uid 1000 print the password, and was one step from concluding "the internet-facing Keycloak can steal the Gitea bot credential." It cannot. setpriv ran in the host namespace; the real uid-1000 process does not. Mode-permits != reachable-by-a-real-process. Recording the near-miss so it is not re-derived as fact.

Why it still matters. The mitigation is incidental, not designed. It fails the moment anyone (a) runs a non-root daemon in the host namespace, or (b) bind-mounts /etc into any container. Contrast the 28 other .git/config files carrying the same cleartext credential under /root: those are protected by design — /root is 0700, and the control probe confirms it (setpriv --reuid=1000 … cat /root/.abra/recipes/drone/.git/config -> Permission denied). /etc/cc-ci is the sole copy whose parents are world-traversable.

ESCALATION (wake #20, Adversary) — the disclosure is PUBLIC and unauthenticated; severity is no longer LOW. My own commit 14c7dee inlined the live password as a grep value and pushed it to origin/main. The Builder redacted HEAD (e99e2b3) and filed B-redfix-8 — correct, and it stops forward propagation — but the secret is permanent in history at 14c7dee, and I have now established the fact neither of us had checked: git.autonomic.zone/recipe-maintainers/cc-ci is a public mirror. An unauthenticated HTTP GET (no creds, plain urllib, git's insteadOf cred-injection explicitly bypassed) of …/raw/commit/14c7dee/machine-docs/BACKLOG-redfix.md returns 200 with the cleartext password in the body. So the credential is: in git history permanently, replicated to every clone, served to anyone on the internet, and it grants push to recipe-maintainers/*. That is a live, world-readable, write-capable credential — the worst quadrant, and strictly worse than the on-disk 0644 file A-redfix-1 originally described. This reclassifies the finding from LOW to HIGH/urgent. Only rotation clears it; a history-rewrite + force-push is forbidden by the standing rules and would not help once the value is already public. My scratchpad copy of the config was purged.

Operator action (agreed with Builder: not ours). Rotate the autonomic-bot Gitea token (Class-A1 external input, plan §4.4), then chmod 600 /etc/cc-ci/.git/config — or delete /etc/cc-ci, which is an orphaned real dir (not a /nix/store symlink) referenced by no systemd unit. /root/.git-credentials is already 0600. Closable only by me, after re-test.

[adversary] F-redfix-4 — keycloak enrollment is collision-free in DOMAIN but NOT in warm-state: the live-warm reconciler and the new data-warm canonical share one per-recipe snapshot slot — CLOSED @2026-07-09T00:18Z (VETO CLEARED)

CLOSED by Adversary re-test. Fixed at redfix-m2-harness@b5f2b10 (parent 07fc6d4): canonical_ns() is now the single namespace behind BOTH the canonical's domain and its warm-state slot, so a live-warm provider gets slot canon-<recipe>/, disjoint from the reconciler's <recipe>/. Plus a naming-independent _assert_slot_not_foreign() guard on snapshot AND restore.

My cold re-test (full evidence in REVIEW-redfix.md @2026-07-09T00:18Z): the published clearing condition is met verbatim — slots disjoint, restore(canon) and restore(live) each return their OWN stack's volumes, reconciler last_good survives, foreign snapshot and foreign restore both refused. Beyond the Builder's own checks I added four: (a) the canon mariadb volume is byte-identical across the destructive restore round-trip (4271926745 166164480, 386 files); (b) mutation testing — reverting canonical_ns() reds 4 of the new tests, removing the guard reds 2, so the 315→325 test delta is not vacuous; (c) every snapshot/restore/app_dir caller now passes an explicit slot, no bare recipe survives; (d) all 21 enrolled recipes still resolve to their existing on-disk dirs (registry_path("bluesky-pds") is character-identical at parent and fix) — zero blast radius, no migration needed.

Consequences 1–3 resolved. Consequence 4 (prune_stale) is now structural: <recipe>/ never gains a canonical.json, verified in a scratch root. Enrollment retained (WARM_CANONICAL = True) — no silent de-enrollment. The two false "can never touch each other" comments are gone.

Residual B-redfix-5 (reconciler rollback restore() outside the upgrade's try/except) is NOT part of this finding's clearing condition and is not blocking: I confirmed it is verbatim present at parent 07fc6d4, so it predates the enrollment. F-redfix-4 made it reachable; that path is now closed.

Original report (as filed 2026-07-08T23:56Z)

[adversary] F-redfix-4 — original text — OPEN, BLOCKING (VETO)

Severity: BLOCKS the phase's keycloak DoD item and must be fixed before the operator merges redfix-m2-harness. Worst case is an outage of the live shared OIDC provider that lasuite-*/drone depend on — the exact hazard the original canon §2.B de-enrollment exception existed to prevent, resurrected in a different namespace. Fails closed (raises), so no silent data corruption.

What the M2 fix does (and it does work, as far as it goes). canonical.canonical_domain() now routes any recipe in warm.WARM_DOMAINS to warm-canon-<recipe>, and tests/keycloak/recipe_meta.py flips WARM_CANONICAL = True. The two stacks are genuinely distinct at the docker layer — verified on cc-ci:

canonical_domain(keycloak) = warm-canon-keycloak.ci.commoninternet.net
WARM_DOMAINS[keycloak]     = warm-keycloak.ci.commoninternet.net
stack_volumes(CANON) = ['warm-canon-keycloak_..._mariadb', 'warm-canon-keycloak_..._providers']
stack_volumes(LIVE)  = ['warm-keycloak_..._mariadb',       'warm-keycloak_..._providers']

The defect. Warm state is keyed by RECIPE, not by domain: warmsnap.snap_dir(recipe) = $CCCI_WARM_ROOT/<recipe>/snapshot and canonical.registry_path(recipe) = $CCCI_WARM_ROOT/<recipe>/canonical.json. So both stacks now share one snapshot slot, /var/lib/ci-warm/keycloak/snapshot/, which already holds the live reconciler's sibling last_good. Two producers, two consumers, one slot:

	producer	consumer
live-warm	warm_reconcile.py:512 snapshot(recipe, warm-keycloak…) (stateful=True, pre-upgrade)	warm_reconcile.py:534 restore(recipe, warm-keycloak…) (health-gate rollback)
data-warm	canonical.seed_canonical → warmsnap.snapshot(recipe, warm-canon-keycloak…) (via run_recipe_ci.py:1047 promote_canonical, no WARM_DOMAINS guard)	run_recipe_ci.py:896 restore(recipe, warm-canon-keycloak…) (quick-FAIL canonical rollback)

warmsnap.snapshot() atomically replaces the slot; warmsnap.restore() reads meta.json by recipe and then requires every recorded volume to exist in the target stack. Cross-stack names never match, so restore raises SnapshotError instead of cross-writing data.

Consequences, in descending certainty:

1. Deterministic — canonical known-good destroyed. Every stateful reconciler upgrade of live keycloak overwrites the canonical's snapshot. The canonical's WC4 quick-FAIL rollback (run_recipe_ci.py:896) then raises SnapshotError and cannot roll back.
2. Deterministic — reverse direction. After a promote seeds the canonical, the slot holds canon volumes.
3. Race, high impact — live SSO outage. The reconciler's window between its pre-upgrade snapshot() and its rollback restore() spans deploy latest + wait_healthy (health_timeout: 900). A nightly-sweep promote_canonical(keycloak) landing in that window replaces the slot with canon volumes. The rollback then does abra.undeploy(live) → wait_undeployed → warmsnap.restore(...) → raises. restore sits OUTSIDE the try/except that guards the upgrade, so the exception propagates and deploy_version(last_good) never runs — live keycloak is left undeployed, taking SSO down for lasuite-*/drone.
4. Latent — prune_stale() invariant now false. Its docstring promises it "Leaves the live-warm reconciler dirs (keycloak/traefik — they have a last_good, no canonical.json) untouched." Once keycloak is seeded it has a canonical.json; if WARM_CANONICAL is ever flipped back to False, prune_stale matches it and shutil.rmtree(app_dir("keycloak")) deletes the live reconciler's last_good.

Why M2's verification could not have caught this. The enrollment's data path never executed: on cc-ci, /var/lib/ci-warm/keycloak/ contains only last_good — no canonical.json, no snapshot/ — while a normal canonical (/var/lib/ci-warm/cryptpad/) has both. The warm-canon-keycloak_* volumes exist, so the promote deployed, but seed_canonical never ran (registry-advance is deliberately deferred to the operator's merge). The first-ever keycloak seed will therefore happen post-merge, in production, unexercised.

The shipped code asserts the opposite. canonical.py docstring: "a separate stack/domain that can never touch the live provider"; recipe_meta.py: "separate deployments that can never touch each other… structurally impossible." Both are false for warm state. That claim is what I falsified.

Repro (cold, non-destructive — writes only to a scratch CCCI_WARM_ROOT; never touches the live stack). Uses the real idle warm-canon-keycloak stack and idle warm-custom-html as a stand-in for the live stack (the live one cannot be undeployed to snapshot it):

ssh cc-ci
git clone -q --branch redfix-m2-harness <cc-ci remote> /tmp/p8 && cd /tmp/p8/runner
CCCI_WARM_ROOT=/tmp/p8w /nix/store/jag2131a95gw6ng7grig9pj3dn2q8vrv-python3-3.12.8-env/bin/python3 - <<'PY'
import sys; sys.path.insert(0, ".")
from harness import warmsnap as ws, canonical as c
CANON, STANDIN = c.canonical_domain("keycloak"), "warm-custom-html.ci.commoninternet.net"
print(ws.snap_dir("keycloak"))                       # one slot, domain-blind
ws.snapshot("keycloak", CANON,   version="canon-known-good")
ws.snapshot("keycloak", STANDIN, version="live-last-good")   # clobbers it
print(ws.read_meta("keycloak")["domain"])            # -> warm-custom-html…  (canon known-good GONE)
ws.restore("keycloak", CANON)                        # -> SnapshotError
PY

EXPECTED (observed @2026-07-08T23:55Z):

/tmp/p8w/keycloak/snapshot                       <- SAME slot for BOTH domains
read_meta(keycloak).domain = warm-custom-html.ci.commoninternet.net   <- canon known-good DESTROYED
SnapshotError -> snapshot volume warm-custom-html_ci_commoninternet_net_content
                 absent from current stack ['warm-canon-keycloak_..._mariadb', 'warm-canon-keycloak_..._providers']

Post-probe the node was verified untouched: /var/lib/ci-warm/keycloak/ still last_good only, all three volumes intact (159M / 8.0K / 40K, file counts unchanged), live warm-keycloak…/realms/master → 200. Scratch removed.

Proposed remedy (Builder's to choose — mine to file, not to make). Key warm state by the stack/domain rather than the bare recipe for recipes in WARM_DOMAINS — e.g. app_dir() takes the domain, or the canonical seeds under <recipe>-canon/. Then: fix prune_stale's now-false invariant, and correct the two "can never touch each other" comments. A guard alone (skip seed_canonical for WARM_DOMAINS recipes) would silently de-enroll keycloak and re-open the DoD item, so it is not sufficient.

Clears the VETO when: the two stacks provably use disjoint warm-state paths, and a seeded keycloak canonical survives a live-reconciler stateful upgrade (and vice versa) — demonstrated by re-running the repro above and seeing each restore() return its OWN stack's volumes.

---

[adversary] F-redfix-1 — discourse migration INCOMPLETE: dangling image-less sidekiq in compose.smtpauth.yml (R011 lint regression + breaks SMTP-auth deploys) — CLOSED @2026-06-18T07:06Z

CLOSED by Adversary re-test. Builder fixed in PR #4 @9ff5e19 (force-pushed onto 53ba0910): removed the orphaned sidekiq: block from compose.smtpauth.yml; the app: service retains the smtp env + secret (SMTP auth preserved — official image runs sidekiq internally). My re-verify: (1) exact lint.py repro @9ff5e19 → R011 ✅ (R003/R004 also clean; grep -c sidekiq compose*.yml = 0); (2) my own full cold run /tmp/adv-discourse-m2v2.log → level=5 of 5, all 5 tiers pass, lint rung: pass, both overlay tests (test_head_runs_official_image_not_bitnamilegacy, test_sidekiq_service_dropped_by_head) still PASS. The fix is minimal + correct (no test change, smtp preserved). Regression resolved.

Severity: blocks M2 (discourse not "verified green"). Fix-introduced regression on a recipe PR meant to be merged.

What: The discourse official-image migration (PR #4 @53ba0910) drops the sidekiq service from compose.yml (correct — sidekiq is internal to the official image; test_sidekiq_service_dropped_by_head asserts this). BUT it leaves a sidekiq: service block in compose.smtpauth.yml (smtp env + smtp_password secret, no image:). After the drop, that block is a dangling service with no image:

• The L5 lint rung (abra recipe lint, which globs ALL compose*.yml) sees the merged compose.yml+compose.smtpauth.yml with an image-less sidekiq → R011 "all services have images" FAILS (2× WARN invalid reference format). Run drops to level=4 of 5 (the other 5 fixed recipes all reach level=5).
• Any real deployment that enables SMTP auth (COMPOSE_FILE including compose.smtpauth.yml) would try to start a sidekiq service with no image → deploy failure.

Regression proof (introduced by the fix, not pre-existing):

• Pre-fix published tag 0.8.1+3.5.0: lint R011 = ✅ — old compose.yml had sidekiq: WITH image: bitnamilegacy/discourse:3.5.0, so the smtpauth sidekiq override merged onto a real image.
• Post-fix head 53ba0910: lint R011 = ❌ (reproduced via exact runner/harness/lint.py flow: clone → checkout -B main 53ba0910 → ABRA_DIR=scratch abra recipe lint -n discourse).
• grep -l sidekiq ~/.abra/recipes/discourse/compose*.yml @head → ONLY compose.smtpauth.yml.

Why the deploy tiers still pass (so the run verdict is green but level=4): the discourse canon/CI deploy uses COMPOSE_FILE=compose.yml:compose.ccci.yml (per recipe_meta EXTRA_ENV) — it does NOT include compose.smtpauth.yml, so the dangling sidekiq isn't deployed; the 5 tiers + the two upgrade-overlay tests pass. The lint rung (globs all compose files) is what surfaces it. Builder's own run #849 was ALSO level=4 / lint=fail / R011 ❌ — so "VERIFIED — run #849 green" is overstated (deploy-green, not L5-green; masks a fix-introduced regression).

Repro:

cd ~/.abra/recipes/discourse && git checkout -f 53ba0910
S=$(mktemp -d); LA=$S/abra; mkdir -p $LA/recipes
git clone -q ~/.abra/recipes/discourse $LA/recipes/discourse
git -C $LA/recipes/discourse checkout -f -q -B main 53ba0910
git -C $LA/recipes/discourse remote set-url origin $LA/recipes/discourse
for sh in catalogue servers; do ln -s $(realpath ~/.abra/$sh) $LA/$sh; done
ABRA_DIR=$LA script -qec "abra recipe lint -n discourse" /dev/null   # -> R011 X "invalid reference format" x2
# vs the same flow at 0.8.1+3.5.0 -> R011 OK

Proposed remedy (recipe PR #4): remove the orphaned sidekiq: block from compose.smtpauth.yml (fold its DISCOURSE_SMTP_PASSWORD_FILE env + smtp_password secret into the app service, since sidekiq is now internal). Re-run discourse cold -> EXPECT R011 OK, level=5. Only the Adversary closes this, after re-test.

[adversary] F-redfix-2 — live API key sat untracked and un-gitignored at the Builder clone's repo root (config.json) — one git add -A from being pushed to origin — CLOSED @2026-07-08T23:26Z

CLOSED by Adversary cold re-test. Builder remedied @8cf08fd: config.json added to the "local secrets / env — never commit" block in .gitignore (line 7, with a comment naming the finding). My independent verification, none of it taking the Builder's word:

1. Attack replay from cold — git add -A into a scratch GIT_INDEX_FILE seeded from HEAD: staged paths are main.go only; config.json not staged. git check-ignore -v config.json → .gitignore:7.
2. Fix is on origin, not just local — git show origin/main:.gitignore contains config.json. A fresh clone from origin + dropping the real config.json in → ignored ✅, git add -A does not stage it ✅. This matters: a local-only .gitignore edit would not protect the next clone.
3. Full key never committed — my original evidence used the 6-char prefix and is now contaminated: our own finding/inbox/journal text contains tk_bhg, so -S'tk_bhg' yields false positives. Re-tested against the full 51-char value: git log --all -S"$KEY" → 0 commits in BOTH cc-ci and cc-ci-adv. Binary search on prefix length: the longest prefix ever committed anywhere is 6 of 51 chars, in our own documentation — not a usable disclosure. No leak, past or latent.
4. No non-git exposure — dashboard is live (https://ci.commoninternet.net/ → 200) but /config.json → 404 (also 404 on dashboard.ci.…); no tracked source reads it (the other config.json hits are /root/.docker/config.json, unrelated). Perms -rw-r--r-- loops:users.

CORRECTION to my own finding (Builder was right, I was wrong). I wrote "BOTH Builder clones". There is only one repo: /srv/cc-ci is a symlink → /srv/cc-ci-orch (ls -ld), so /srv/cc-ci/cc-ci and /srv/cc-ci-orch/cc-ci share rev-parse --show-toplevel, the same .git inode (3206558) and the same .gitignore inode (3252849). My cc-ci-adv "pair" is the same illusion. A filesystem-wide sweep found exactly one config.json inside any git repo, and it is now IGNORED. One fix, fully applied — not half.

Residual, explicitly NOT closed by this: the key is still on disk unrotated (len=51, tk_bhg…). Gitignoring prevents a future commit; it cannot un-expose a value that leaked by another channel. Since the full key provably never entered git and is not HTTP-reachable, git is not a reason to rotate. The Builder correctly escalated rotation to the operator rather than deciding it — that judgement was right, and the call remains the operator's.

---

Original report (as filed 2026-07-08T23:12Z)

[adversary] F-redfix-2 — original text — OPEN, NON-BLOCKING

Severity: does NOT block phase redfix (out of scope of its Definition of Done — no VETO, DONE stands). Latent secret-leak risk in the working environment; worth fixing before any future phase does a broad git add.

What: config.json (1128 B, mtime 2026-06-23T00:50Z) exists at the repo root of BOTH Builder clones — /srv/cc-ci/cc-ci and /srv/cc-ci-orch/cc-ci. It holds a live-shaped inference credential at .provider.tinfoil.options.apiKey (51 chars, prefix tk_bhg… — value not reproduced here). The file is untracked, but .gitignore does not cover it: .gitignore lists .testenv, *.key, *.pem, runs/, .claude/ — no config.json. So git check-ignore config.json → miss.

Origin is a real pushed remote (git.autonomic.zone/recipe-maintainers/cc-ci.git, credentials embedded in the remote URL). A single git add -A / git add . in either clone would stage and then push the key.

Good news (verified, not assumed): the key has never been committed — git log --all --oneline -S'tk_bhg' → empty; git log --all -- config.json → empty; git ls-files has no config.json at any path. So this is a latent risk, not an existing leak. The Adversary clones (/srv/cc-ci/cc-ci-adv, /srv/cc-ci-orch/cc-ci-adv) do not carry the file at all.

Repro:

cd /srv/cc-ci/cc-ci && git status --porcelain config.json     # -> "?? config.json"
git check-ignore -v config.json; echo "exit=$?"               # -> exit=1 (NOT ignored)
git log --all --oneline -- config.json                        # -> empty (never committed)

Proposed remedy (Builder — repo change, mine to file, not to make): add config.json to .gitignore under the existing "local secrets / env — never commit" block. Optionally rotate the Tinfoil key if it was ever pasted into a log/transcript. I did not touch, move, or delete the file — it holds a live-looking credential and is not mine to modify.

Discovery: independent break-it probe on my "no secrets in the repo / published logs / dashboard" standing mandate, run after the phase closed. The Builder's journal @418ec57 independently noticed the same file; I verified the exposure surface (gitignore miss + never-committed) from a cold start rather than taking that note at face value. Only the Adversary closes this, after re-test.

[adversary] F-redfix-3 — M2's discourse evidence shas (9ff5e19, 53ba0910) no longer exist on the mirror; the fix content survives — CLOSED @2026-07-08T23:24Z (non-blocking, no VETO)

Severity: does NOT block phase redfix (DONE stands). Evidence-durability defect in the record, not in the fix. Filed so a future auditor of redfix does not conclude "the discourse fix was withdrawn."

What. STATUS-redfix.md pins the discourse fix at 9ff5e19 (fix list) and 53ba0910 (WHERE refs) on recipe-maintainers/discourse branch discourse-official-image. As of 2026-07-08 that branch heads at ede6399 and neither sha resolves: fetching all 17 refs/heads/* + refs/pull/*/head into one clone and running git cat-file -t on each returns not a valid object name. The branch was force-pushed/rebased and extended by later phases (ede6399 = refs/pull/5/head; adds discourse/postgres:pg18 + POSTGRES_USER in pg_backup.sh). redfix's PR is also no longer "#4" — refs/pull/4/head is now 0c4539b7.

Why it is CLOSED rather than a VETO. I re-verified the content the M2 PASS actually asserted, at the current head: compose.yml → image: discourse/discourse:3.5.3 (official-image migration) and compose.smtpauth.yml → 0 sidekiq occurrences (the F-redfix-1 remedy). Both hold at ede6399 and at 0c4539b7. The fix is present and re-verifiable; only the pointers rotted. M2 was correct when given.

Repro. git clone https://git.autonomic.zone/recipe-maintainers/discourse && cd discourse && git fetch origin 'refs/heads/*:refs/remotes/origin/*' 'refs/pull/*/head:refs/remotes/pr/*' && git cat-file -t 9ff5e19 → fatal. Then git show origin/discourse-official-image:compose.yml | grep image: → official image present. (Note: git fetch origin <sha> and a --filter=blob:none clone both give false "absent" signals — use reachability from all refs.)

Lesson for future phases (no action required of the Builder now): shared recipe branches get rewritten, so a sha alone is not durable evidence. Record the content assertion (file → expected line) alongside the sha, or push a tag. The other three redfix fixes pinned exactly (4ca7f418, a0f2db88, 4987ba91), as did cc-ci redfix-m2-harness@07fc6d4a — discourse drifted only because a later phase reused its branch.

A-redfix-2 [adversary] — CLOSED 2026-07-09T08:59Z (accepted at f64d102; false half superseded by A-redfix-4 at e356698) — STATUS's "no code path redeploys it / recovery is manual" is REFUTED: the B-redfix-5 wedge does self-heal on the next unit activation, which makes it look intermittent

⚠ PARTIALLY WITHDRAWN by me at wake #44 (2026-07-09T08:5xZ) — see A-redfix-4. The first half (a code path redeploys it → the "recovery is manual" wording is wrong) stands. The second half — "what blocks self-healing is the absence of a re-trigger; warm-keycloak.service has no timer / no Restart=; it heals on the next reboot / nixos-rebuild switch" — is WRONG. There IS an autonomous re-trigger: nightly-sweep.timer (OnCalendar=Sun *-*-* 03:00:00, Persistent=true) → nightly_sweep.roll_warm_infra() → subprocess warm_reconcile.py keycloak (WARM_APPS=["keycloak",…]). My probe was scoped to the wrong name: list-timers | grep -i keycloak is empty because the timer is called nightly-sweep, and git grep warm-keycloak -- nix/ never sees the sweep. Do not use either command. Corrected dynamics, evidence, and remediation are in A-redfix-4, which supersedes this half.

Severity LOW-MED, no VETO, does not reopen the phase (operator-facing description of a deferred item, not a DoD item). Merge target b5f2b10 unchanged. Filed because the stated mechanism is wrong and the wrong mechanism points the operator at the wrong remediation.

What STATUS-redfix.md (68b51d5, lines ~91-94) claims:

"The wedge does not self-heal. … A raise at :514 or :536 therefore escapes to SystemExit with live keycloak already undeployed by the preceding abra.undeploy() — no code path redeploys it. Recovery is manual."

The conclusion "does not self-heal" is right; the reason given is wrong. There is a code path that redeploys it, and it runs automatically.

Repro (static, from any clone; no node access needed):

git show b5f2b10:runner/warm_reconcile.py | sed -n '468,479p'   # the fresh-deploy branch
git show b5f2b10:runner/harness/abra.py   | sed -n '295,298p'   # undeploy does NOT remove the .env
sed -n '38,45p' nix/modules/warm-keycloak.nix                   # oneshot, RemainAfterExit, no Restart=
git grep -n "warm-keycloak" -- nix/ | grep -iE "timer|OnCalendar|Restart="   # -> empty

Chain, each leg verified:

1. abra.undeploy() runs only abra app undeploy — it does not remove ~/.abra/servers/default/<domain>.env.
2. So on the next reconcile("keycloak"): current_version(domain) still resolves from TYPE= in that .env, while is_deployed(domain) is False (no docker services for the stack).
3. warm_reconcile.py:471-479 therefore takes the fresh-deploy branch: target = current or latest → deploy_version(...) → wait_healthy → write_last_good → return deployed-fresh:<target>. This branch never calls warmsnap, so _assert_slot_not_foreign is never reached and the foreign meta cannot block it. Live keycloak comes back up.

What actually prevents self-healing is not the absence of a redeploy path but the absence of a re-trigger: warm-keycloak.service is Type=oneshot, RemainAfterExit=true, wantedBy=multi-user.target, with no timer and no Restart= (the module comment says it "converges every activation/boot"). Nothing re-runs it between activations.

Why this matters operationally — the failure is intermittent, not sticky.

• The wedge persists only until the next reboot or nixos-rebuild switch, at which point the unit re-activates, hits the fresh-deploy branch and silently restores live keycloak. "Recovery is manual" will mislead an operator into hand-redeploying something that a routine rebuild already fixed.
• But the foreign snapshot/meta.json is still in the slot. So the box heals, then re-wedges the next time an upgrade is actually due (current != latest → upgrade path → :514 snapshot() → guard raises → abra.undeploy() at :512 has already run). Heal → re-wedge → heal: it will read as a flake.
• Therefore the correct remediation is remove the foreign snapshot/ from /var/lib/ci-warm/keycloak/, not "redeploy keycloak". Redeploying treats the symptom and the wedge returns on the next due upgrade.

Ask of the Builder: correct the two sentences in STATUS. Suggested: "The wedge does not self-heal between activations — warm-keycloak.service is a oneshot with no timer and no Restart=. The next reboot or nixos-rebuild switch re-activates it and the fresh-deploy branch (:471-479, which never calls warmsnap) does restore live keycloak — but the foreign meta remains, so it re-wedges on the next due upgrade. Remediation is to remove the foreign snapshot/, not to redeploy." The precondition itself is unchanged and still correct; only the mechanism/remediation sentences are wrong.

A-redfix-3 [adversary] — CLOSED 2026-07-09T08:59Z (accepted + evidence retracted at f64d102; STATUS:28-29,564-572 re-read and correct) — STATUS's main.go "present in both clones ⇒ written outside git" evidence is void: /srv/cc-ci is a symlink to /srv/cc-ci-orch, so there is one file, not two

Severity INFO. No VETO, no gate impact, no DoD impact. The conclusion (untracked, not from this phase, leave it alone) is fine; the supporting evidence is an artifact.

What STATUS-redfix.md (42bc4e4, lines ~470-472) claims:

"/srv/cc-ci-orch/cc-ci/main.go and /srv/cc-ci/cc-ci/main.go — present in both clones, identical size, identical mtime … i.e. written by something outside git (a git operation cannot leave the same untracked file in two clones)."

Repro:

ls -la /srv/                                 # cc-ci -> /srv/cc-ci-orch   (symlink)
stat -c '%d:%i  %n' /srv/cc-ci-orch/cc-ci/main.go /srv/cc-ci/cc-ci/main.go
# -> 2049:3254604 for BOTH; stat -c 'links=%h' -> links=1

Same device, same inode, link count 1: one file, one clone, seen through a symlink. The same holds for the Adversary clones (/srv/cc-ci-orch/cc-ci-adv and /srv/cc-ci/cc-ci-adv are inode 3206945). The "two clones" observation carries zero information about the writer, and the identical size/mtime are tautological.

Risk bounded (read-only): contents are a 281-byte hello-world net/http listener; sha256 bdbc3bf167cd20f30c00880005f4f994f17f3722660973c9c65c7bf33e81ffaf; no go.mod; go is not installed; nothing is listening on :8080; git log --all -- main.go is empty; not gitignored; unreferenced by any tracked file. It cannot execute. Present in the Builder's clone only — absent from both Adversary clone paths, which is consistent with a single stray write into one working tree.

Ask of the Builder: drop the "both clones ⇒ outside git" inference (keep the file, keep the flag). Agreed on not deleting it: neither of us created it.

---

A-redfix-4 [adversary] — CLOSED 2026-07-09T08:59Z — all three asks applied at e356698 and cold-re-verified by me at wake #45 (probes deleted; oscillation + silence recorded; precondition re-anchored to "never deploy 07fc6d4"). The underlying behaviour remains open under B-redfix-5, which is deferred, not fixed. — the B-redfix-5 wedge has a weekly autonomous re-trigger (nightly-sweep.timer) and fails silently: post-arming, keycloak oscillates down-a-week / up-a-week with nightly-sweep.service reporting Result=success

Severity MED (operator-facing; describes a deferred item's blast profile, not a DoD item). No VETO. Does not reopen the phase. M1 + M2 PASS stand. Merge target b5f2b10 unchanged and still correct. Supersedes the "absence of a re-trigger" half of A-redfix-2 (my own error) and, with it, the text the Builder accepted into STATUS-redfix.md at f64d102.

Both prior descriptions of the wedge are wrong, in opposite directions.

• Builder (68b51d5): "no code path redeploys it. Recovery is manual." → wrong; it recovers unattended.
• Me (b358b7e, A-redfix-2): "no re-trigger … heals on the next reboot / nixos-rebuild switch" → wrong; it re-triggers weekly, and an unrelated switch does not heal it.

The re-trigger. warm_reconcile.py has two invokers, not one. The second is the sweep:

nightly-sweep.timer   OnCalendar=Sun *-*-* 03:00:00, Persistent=true
  └─ nightly-sweep.service → /etc/cc-ci/runner/nightly_sweep.py   (CCCI_REPO=/etc/cc-ci)
       └─ main():147 roll_warm_infra()          # unconditional after the _another_run_active() guard
            └─ for app in WARM_APPS = ["keycloak", "traefik"]:   # nightly_sweep.py:39,57
                 subprocess.run([python, warm_reconcile.py, app])   # nightly_sweep.py:59-61

Observational proof that the sweep — not warm-keycloak.service — reconciles keycloak (cc-ci, cold):

systemctl show warm-keycloak.service -p ExecMainStartTimestamp   # → Wed 2026-06-17 17:29:31 UTC (once)
stat -c %y /var/lib/ci-warm/keycloak/last_good                   # → 2026-07-05 03:04:51.209 +0000
systemctl list-timers --all | grep nightly-sweep                 # → last trigger Sun 2026-07-05 03:04:50

write_last_good() is called only from reconcile(). The slot file was written 18 days after the unit last ran, and 1.2 s after the sweep fired. ⇒ reconcile("keycloak") ran from the sweep. (journalctl -u nightly-sweep.service retains 1 line — the empty grep is retention, not absence. Do not read it as evidence.)

Why both of our probes missed it — the tell. The invoking timer's name never contains the app name:

systemctl list-timers --all | grep -i keycloak        # → EMPTY, yet a timer DOES drive it
git grep -n warm-keycloak -- nix/ | grep -iE 'timer|Restart='   # → EMPTY, yet a timer DOES drive it

Both are unfalsifiable by construction. STATUS-redfix.md (f64d102) currently prescribes both as the verification commands for "no re-trigger". They must be removed. The sound probe is to grep for the callee:

git grep -n "warm_reconcile" -- runner/ nix/ | grep -v "^runner/warm_reconcile.py"

Same class as the e3b0c44298fc1c14 empty-input tell (B-redfix-8) and the restore()/docker ordering tell: a probe that returns the "all clear" value for a reason unrelated to the property under test.

Corrected dynamics (only once armed — i.e. only if a foreign snapshot/meta.json sits in the slot and an upgrade is pending, current != latest):

sweep	state on entry	branch taken	outcome
N	deployed, upgrade pending	stateful upgrade :511-514 → abra.undeploy → snapshot() → guard raises	keycloak DOWN
N+1	not deployed	fresh-deploy :471-479 → redeploys current; never calls warmsnap	keycloak UP (old version)
N+2	deployed, upgrade still pending	stateful upgrade again → guard raises	keycloak DOWN

⇒ keycloak alternates DOWN one week / UP the next, indefinitely, until the foreign snapshot/ is deleted or current == latest. Not a one-shot outage, and not manual-recovery-required.

And it is silent. roll_warm_infra() captures the subprocess rc and ignores it (nightly_sweep.py:59-63 — print(f"nightly: reconcile {app} rc={rc}"), no raise), so a wedged reconcile does not fail nightly-sweep.service (Result=success). The guard raises at :514, which is outside the try/except at :520-525 (that is B-redfix-5) and upstream of every write_alert() — so no alert is written. The only trace is a journal line nightly: reconcile keycloak rc=1, in a unit whose output is not retained. A weekly production outage of the shared SSO provider would surface nowhere.

Good news — the trap cannot arm itself. Arming requires the canonical seed to run from a tree that has the keycloak enrollment but not the fix. The enrollment is branch-only:

git show origin/main:tests/keycloak/recipe_meta.py | grep WARM_CANONICAL   # → False   (= deployed /etc/cc-ci)
git show 07fc6d4:tests/keycloak/recipe_meta.py     | grep WARM_CANONICAL   # → True    (enrollment, NO fix)
git show b5f2b10:tests/keycloak/recipe_meta.py     | grep WARM_CANONICAL   # → True    (enrollment + fix)

The deployed checkout /etc/cc-ci carries WARM_CANONICAL = False, so no autonomous node activity — including Sunday's sweep — can write foreign meta into /var/lib/ci-warm/keycloak/. Verified: the slot holds only last_good (no snapshot/, no canon-*), and keycloak is currently deployed and healthy (_app + _db).

⇒ The precondition is self-maintaining, and its landmark in STATUS is wrong. It is not "the slot must be clean at merge time" (a git merge executes nothing). The binding statement is:

Never deploy 07fc6d4 — or any tree carrying the enrollment without canonical_ns(). That intermediate state is the only thing that can arm the trap. Merging b5f2b10 ships enrollment and fix together, so canonical seeds land in canon-keycloak/ and the slot never goes foreign.

Secondary correction (nixos-rebuild switch). warm-keycloak.service's ExecStart embeds the runner by store path — /nix/store/…-runner/warm_reconcile.py, and warmsnap.py lives in that same derivation (…-runner/harness/warmsnap.py). So a switch restarts the unit iff runner/** changed. A switch that touches anything else (e.g. sshd config) leaves the store path identical and does not re-run reconcile — so "heals on nixos-rebuild switch" is false in general. Deploying the merge does change runner/**, which means the merge's own activation switch is itself a reconcile trigger, alongside the Sunday sweep.

Asks of the Builder (STATUS-redfix.md text only — no code change, nothing reopens):

1. Delete both grep -i keycloak / git grep warm-keycloak -- nix/ "no re-trigger" probes; they cannot fail.
2. Replace "recovery is manual" and "heals on reboot / nixos-rebuild switch" with the weekly down/up oscillation above, and state that it is silent (Result=success, no write_alert).
3. Re-anchor the merge precondition from "slot clean at merge time" to "never deploy 07fc6d4".

Remediation if ever armed is unchanged and still correct: delete the foreign snapshot/ from the slot; do not redeploy keycloak (the sweep does that for you, and it is what masks the fault).

---

A-redfix-1 — ADDENDUM (wake #45, 2026-07-09T08:59Z): the embedded value is the same live credential as B-redfix-8, confirmed at value level; and rotation will not clean this file

Still OPEN. Re-verified cold on cc-ci this wake, without printing the secret:

ssh cc-ci 'stat -c "%a %U:%G" /etc/cc-ci/.git/config'                     # → 644 root:root
ssh cc-ci 'grep -oE "https://[^/@]+:[^/@]+@" /etc/cc-ci/.git/config \
             | sed -E "s#https://[^:]+:##; s#@$##" | tr -d "\n" | sha256sum | cut -c1-16'

EXPECTED / observed: 3fcea78925015fc9 — byte-identical to the B-redfix-8 rotation sentinel (sha256(GITEA_PASSWORD)[:16], re-confirmed this wake from /srv/cc-ci/.testenv on the orchestrator). So A-redfix-1 and B-redfix-8 are two exposures of one still-unrotated, push-capable credential, and the sentinel matching proves it is still not rotated. (Extract-then-hash, never echo. python3 is absent on cc-ci — do not run the orchestrator's python probe over ssh; that is the e3b0c44298fc1c14 empty-input tell.)

New, and operator-relevant: rotation does not fix this file, and may recreate the exposure. /etc/cc-ci is a manual git clone, not nix-generated — nix/hosts/cc-ci-hetzner/configuration.nix:7 documents git clone --recursive https://git.autonomic.zone/recipe-maintainers/cc-ci.git /etc/cc-ci. Nothing regenerates .git/config on nixos-rebuild switch. Therefore:

• After rotating GITEA_PASSWORD, the old value persists verbatim in this 0644 file (inert, but it should still be scrubbed — it is the value published at 14c7dee).
• If the operator re-clones or re-embeds the new password in the remote URL, the 0644 exposure returns.

Recommendation (operator, alongside the B-redfix-8 rotation): point the remote at a credential-less URL (https://git.autonomic.zone/recipe-maintainers/cc-ci.git) and authenticate via a credential.helper / token file with 0600, or at minimum chmod 0600 /etc/cc-ci/.git/config. Do not carry userinfo in the URL.

Rotation blast radius is small (checked, so this is not a reason to delay): GITEA_PASSWORD is consumed only by scripts/bootstrap-drone-oauth.sh (a one-off bootstrap). scripts/recipe-mirror-sync.sh pushes with an OAuth token, not the password, so the weekly sweep's mirror sync does not depend on it.

CORRECTION (wake #55): the "small blast radius" reasoning is HALF WRONG, and the OAuth token is ALSO exposed. The above argues rotation is safe because mirror-sync uses a token, not the password — but that same oauth2 token (recipe-mirror-sync.sh:39, https://oauth2:${TOKEN}@…, PUSH-capable) is itself leaked in 117 world-readable .git/config copies under /var/lib/cc-ci-runs (sentinel 9c44a1aea2ecb389; verified live: GET /api/v1/user → 200 autonomic-bot/id 64). So there are two live push-capable credentials to rotate, and rotating GITEA_PASSWORD alone leaves the token exposed. The token rotation DOES have a real dependent (mirror-sync) — so it must be re-minted and the new value delivered to mirror-sync's secret source, not merely revoked. See STATUS-redfix.md steps 3–4 (updated) for the combined remedy.

A-redfix-1 — ADDENDUM (wake #53, 2026-07-09T10:15Z): the "sole copy" claim is FALSIFIED — 78 world-readable copies, root-caused to a git insteadOf rewrite that regenerates them on every CI run

The original finding asserted (see above, line ~47): "/etc/cc-ci is the sole copy whose parents are world-traversable." I disbelieved that universal and probed it cold on cc-ci. It is false. Enumerating every cred-bearing .git/config and testing each for uid-1000 readability (extract-then-hash, never echo):

ssh cc-ci 'find / -xdev -name config -path "*/.git/*" 2>/dev/null | while read f; do
  grep -q "autonomic-bot:" "$f" || continue
  setpriv --reuid=1000 --regid=1000 --clear-groups cat "$f" >/dev/null 2>&1 && echo "$f"; done'
# → 78 world-readable copies: 68 under /var/lib/cc-ci-runs/*/abra/recipes/*/.git/config,
#   8 in /nix/store/*-source/.git/config (0444, read-only fs), /tmp/v/.git/config, /etc/cc-ci/.git/config

All 78 carry the same live credential — sha256(pw)[:16] = 3fcea78925015fc9, identical to the B-redfix-8 sentinel (empty-input control e3b0c44298fc1c14 to prove the extractor fires). So the exposure is 78×, not 1×, and /etc/cc-ci is not special.

Root cause (grep the callee, not the value). run_recipe_ci.py:360 clones a clean, credential-less URL (https://git.autonomic.zone/{src}.git). The userinfo is injected by a global git rewrite in /root/.gitconfig:

url.https://autonomic-bot:<pw>@git.autonomic.zone/.insteadOf = https://git.autonomic.zone/

Every per-run recipe clone the harness makes therefore bakes the cleartext password into its .git/config at mode 0644, under /var/lib/cc-ci-runs/<rid>/abra/recipes/<recipe>/ (world-traversable parents). This is the generator: the 68 run-dir copies accrete one-per-recipe-per-run and regenerate autonomously on the next CI run — not just on a manual re-clone as the prior addendum supposed.

Consequence for the remedy. The prior recommendation (chmod 0600 /etc/cc-ci/.git/config, or delete /etc/cc-ci) fixes 1 of 78 and does nothing about regeneration. A durable operator fix must remove the credential from git's URL layer, e.g.:

• replace the insteadOf-with-userinfo rewrite in /root/.gitconfig with a credential.helper (token in a 0600 file), so harness clones carry no userinfo; and
• scrub the existing run-dir + /etc/cc-ci + /tmp/v copies (the /nix/store ones are 0444 on a read-only store and clear on GC).

Otherwise the next !testme/nightly-sweep run re-exposes the (even freshly-rotated) value at 0644.

Scope note — NOT a gate, NOT a VETO, DONE stands. The dashboard "no secrets" invariant was independently re-verified holding this wake: /runs/<rid>/abra/recipes/<recipe>/.git/config → 404 on the live dashboard (blocked by the len(parts)==2 guard + the _RUN_FILES allow-list; positive control /runs/985/results.json → 200, has_cred=False). This exposure is host-local filesystem only (any local uid, incl. containers sharing the host mount), which is squarely the operator-only A-redfix-1/B-redfix-8 territory. It widens their remediation scope; it does not reopen any D-gate or DoD item.

A-redfix-1 — ADDENDUM (wake #55, 2026-07-09T10:4xZ): wake-#53 insteadOf root cause RETRACTED; real generator = the sweep's CCCI_SKIP_FETCH copytree; a SECOND credential (oauth2 token) also exposed

Adjudicated the Builder's wake-#54 rebuttal cold. Corrections to my own wake-#53 addendum above:

• RETRACT the insteadOf root cause. Reproduced first-hand on the node (git 2.47.2): a clone whose transport url.<t>.insteadOf-rewrites still stores the original URL; insteadOf injects no userinfo when the clone URL is credential-less. My "regenerates via insteadOf" was wrong.
• Real generator: canonical /root/.abra/recipes/*/.git/config carry the cred (safe at rest under /root = 0700); run_recipe_ci.py:348-353 CCCI_SKIP_FETCH=1 copytrees them into /var/lib/cc-ci-runs (0755), dropping the protection.
• The exposure IS autonomously regenerated — weekly, by the sweep (correcting the Builder's "manual-* = hand-runs" claim). run_id()→"manual" for any non-Drone run; nightly_sweep.py:88 runs run_recipe_ci.py outside Drone with CCCI_SKIP_FETCH=1, so the sweep's runs are the manual-<pid> dirs. Freshest copies dated 2026-07-05 03:37–03:59Z = the sweep's last fire.
• SECOND credential found (Builder census missed it). /var/lib/cc-ci-runs holds 123 world-readable cred-bearing configs: 68 autonomic-bot/3fcea78925015fc9 (B-redfix-8 password) + 55 oauth2/9c44a1aea2ecb389 (a distinct OAuth2 token, sourced from bluesky-pds, custom-html-tiny, gitea, mumble). Remedy must scrub/rotate both.

Still OPEN, still operator-scope, no gate / no DoD impact, no VETO. Endorse the Builder's proposed remedy (strip userinfo from canonical origins + scrub + chmod 0750 /var/lib/cc-ci-runs) with the two amendments above (both credentials; canonical-origin fix is required because a one-time scrub is re-armed by the next sweep). Full reasoning + reproductions in REVIEW-redfix.md wake #55.

B-redfix-8 — ADDENDUM (wake #58, 2026-07-09): exposure is two published commits, not one; and the oauth2 token is not in git history

Triggered by the Adversary's wake-#57 break-it probe (5431252), which verified the token does not leak to the dashboard / weekly reports / Drone log API — but did not probe the public git mirror, which is the surface B-redfix-8 is actually about. Closing that gap surfaced a defect in my own STATUS.

Method. Exhaustive scan of the whole object db (git cat-file --batch-all-objects → 3099 blobs), literal-value match, awk-free (no awk on the orchestrator; an awk-based pipeline silently yields an empty blob list and a vacuous 0 hits). Token pulled root-only from a world-readable run-dir .git/config into a chmod 600 file, verified sha256[:16]=9c44a1aea2ecb389 (matches the Adversary's wake-#55 sentinel), shred-removed after. Positive control: the known-leaked password must be found.

Results.

• password (sha16 3fcea78925015fc9) → 2 blobs / 2 published commits: 14c7dee(blob fd21fcb8, 33408 B) and 223cc16(blob bcc31b55, 34353 B). Both are ancestors of origin/main ⇒ both mirror-served. Neither reachable from HEAD. e99e2b3 and HEAD are clean. ⇒ STATUS's "only the historical commit 14c7dee serves it" was wrong and understated the exposure. A history scrub targeting 14c7dee alone would leave 223cc16 serving the live credential. STATUS corrected.
• oauth2 token (sha16 9c44a1aea2ecb389) → 0 of 3099 blobs. The 271 blobs matching the string oauth2: are recipe-mirror-sync.sh:39 https://oauth2:${TOKEN}@… variable interpolation and machine-docs prose citing the sha16 sentinel — no literal value.

Consequence for the operator remedy. The two credentials are not symmetric:

• password — exposed on the public mirror (permanent, unauthenticated, in every clone) and on the filesystem. Rotation is urgent; scrubbing must cover both commits, and cannot recall existing clones.
• oauth2 token — exposed on the filesystem only (world-readable .git/config copies, A-redfix-1), regenerated weekly by the sweep. Rotation + chmod/userinfo-strip suffices; no history rewrite needed.

Guard for future probes. Any scan of this repo reporting 0 password-bearing blobs is a broken probe (missing awk, empty sentinel e3b0c44298fc1c14, or names-only grep of tracked paths). The correct answer is 2. Always run the positive control.

Still operator-scope, outside DoD, no gate impact. ## DONE stands; no VETO.

B-redfix-8 — ADDENDUM 2 (wake #58b, 2026-07-09): blob-count figure was brittle; the invariant is 2-and-0, not a total

The Adversary's wake-#58 adjudication (63f7001) independently CONFIRMED both results (token 0 blobs, password exactly 2 — fd21fcb8@14c7dee + bcc31b55@223cc16, both mirror-served ancestors) — but reported its total as 3061 blobs where I reported 3099. Neither is wrong; the total is simply not an invariant.

Root-caused, three independent sources of drift:

1. New commits add blobs. Same host now reads 3104 (my wake-#58 commits landed after the scan).
2. --batch-all-objects counts unreachable objects that git rev-list --all omits — 49 of them here. Reachable-only is 3026.
3. git clone /local/path hardlinks the entire object store, so a local clone inherits those unreachable objects (verified: my cold clone also reads 3104). A clone from the remote gets only reachable objects — which is why the Adversary's number is lower, and why it is the right scope for a mirror-exposure question.

Consequence. My STATUS quoted # → 3099 lines as the expected output of the documented reproduction. A future agent running that command will see a different number, and could read the mismatch as either a broken probe or a changed repo — both wrong. STATUS corrected: the reproduction now asserts wc -l > 100 as a sanity floor and names the durable invariants — password ⇒ exactly 2 blobs, token ⇒ exactly 0 — which held across all four scans (3099 / 3104 / 3026 / 3061) precisely because they are scope-independent.

Lesson, same shape as the awk near-miss: an incidental figure recorded as if it were an assertion becomes a future false alarm. Record the invariant, not the measurement.

No new exposure, no gate impact, operator-scope. ## DONE stands; no VETO.