recipe-maintainers/cc-ci
0
0
Fork 0
Code Issues Pull Requests 1 Actions Packages Projects Releases Wiki Activity
Files
bb70963f07db14f3ea5987268744b33b9513a4b2
cc-ci/machine-docs/BACKLOG-redfix.md

757 lines
59 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

# BACKLOG — phase `redfix`
## Build backlog
### M1 — investigate + isolate + classify (all six)
- [ ] discourse — reproduce cold-deploy timeout/wedge in isolation; root-cause (headroom vs
convergence bug vs upstream compose defect `sidekiq.depends_on: discourse`); classify.
- [ ] mattermost-lts — `test_restore.py::test_restore_returns_state` in isolation: green→load flake,
red→diagnose restore (recipe vs test).
- [ ] mumble — `custom/test_protocol_handshake.py::test_handshake_completes_with_channel_presence` in
isolation (canonical already present from today → likely flake; confirm).
- [ ] bluesky-pds — warm-canonical promote routing: why `warm-bluesky-pds…` → 000 over HTTPS while
container healthy internally + cold-test domain routes. Find cc-ci warm-machinery defect.
- [ ] gitea — `3.5.3→3.6.0` warm advance crash (`app.ini` read-only, JWT save). Recipe vs harness.
- [ ] keycloak — de-enrolled (live-warm OIDC collision). Design collision-free warm domain/namespace.
### M2 — FIX + verify all six (recipe PR or harness improvement)
**Execution gated on M1 PASS** (avoid node contention with Adversary M1 re-runs; classifications must
hold). Concrete fix designs from M1 evidence:
- [ ] **mattermost-lts** (recipe PR, clearest) — add `pg_backup.sh` (immich pattern, no VectorChord
bits): `backup(){ pg_dump -U mattermost mattermost | gzip > /var/lib/postgresql/data/backup.sql; }`
`restore(){ gunzip -c …/backup.sql | psql -U mattermost -d mattermost -f -; }`. compose: add
`configs: pg_backup → /pg_backup.sh`; postgres labels → `backup.pre-hook: /pg_backup.sh backup`,
`restore.post-hook: /pg_backup.sh restore`, `backup.volumes.postgres.path: backup.sql` (dump-only,
drop the whole-PGDATA `backup.path` + the `rm` post-hook). Verify via `!testme` → restore green.
- [ ] **bluesky-pds** (recipe PR) — eliminate the `app`-alias collision on shared proxy: give the PDS
service a unique name (e.g. `pds`) OR a unique network alias, and update caddy refs
(`reverse_proxy`, `on_demand_tls ask http://…/tls-check`), healthcheck, backup labels, ops/test
service= refs. Verify warm promote → 200 on /xrpc/_health. (NOTE: cc-ci harness `ops.py`/tests
reference `service="app"` for bluesky? check + update if the recipe service renames — but recipe
mirror is PR-only; cc-ci-side refs are a separate cc-ci change.) Confirm exact approach in M2.
- [ ] **gitea** (recipe PR) — make app.ini writable on the warm-reattach advance so 3.6.0 can persist
the JWT secret: render app.ini into the WRITABLE `config:/etc/gitea` volume via the existing
`docker-setup.sh` entrypoint (copy the templated config to a writable path) instead of the
read-only `app_ini` docker-config mount; OR ensure the persisted JWT secret is accepted without
rewrite. Verify the 3.5.3→3.6.0 advance promotes. (Ties to LFS PR #1.)
- [ ] **keycloak** (harness, cc-ci branch) — `canonical.canonical_domain(r)`: return a collision-free
domain when `r` is a live-warm provider (`r in warm.WARM_DOMAINS`) → e.g.
`warm-canon-<r>.ci.commoninternet.net`; else keep `warm-<r>` (zero blast radius on the 15 others).
Set keycloak `WARM_CANONICAL=True`. Verify keycloak promotes at warm-canon-keycloak WITHOUT
disrupting live warm-keycloak (200 throughout).
- [ ] **mumble** (harness, cc-ci branch) — stabilize the handshake under load: add a READY_PROBE/
readiness gate (TCP 64738 stably listening + a successful handshake) before the custom tier
and/or raise `retry_handshake` budget; verify green under a concurrent-load re-run.
- [ ] **discourse** (TRICKIEST — decide in M2) — the overlay `test_upgrade.py` asserts a
bitnamilegacy→official migration absent from all releases/main. Options: (a) cc-ci test PR
(--with-tests) scoping the faithfulness assertion to ONLY fire when the head actually performs
the migration (image still bitnamilegacy → N/A, not RED) — NOT a weakening, a correct scope; +
file an upstream recipe issue/PR for the real bitnamilegacy→official migration. (b) recipe PR
doing the migration (major rewrite — official discourse image is launcher-based, likely
infeasible cleanly). Lean (a)+tracked-upstream; may need operator input (DEFERRED?) — assess in M2.
### M3 — post-VETO remediation (F-redfix-4)
- [x] **keycloak warm-state slot collision** — FIXED at `redfix-m2-harness`@`b5f2b10`. `canonical_ns()` is
now the one namespace behind both the canonical's domain and its warm-state slot; live-warm provider
→ `canon-<recipe>` slot, disjoint from the reconciler's `<recipe>/`. Plus a naming-independent
`_assert_slot_not_foreign()` guard. Unit suite 315→325; clearing condition re-run green on cc-ci
(each `restore()` returns its own stack's volumes; reconciler `last_good` survives). Verify per
STATUS-redfix.md "Gate: M2 RE-CLAIMED".
- [ ] **B-redfix-5 — reconciler's post-`undeploy` warmsnap calls are outside the upgrade's `try/except`**
(NOT blocking; NOT part of F-redfix-4's clearing condition; recorded so it is not silently dropped).
In `warm_reconcile.py` there are **two** such sites, both after `abra.undeploy(domain)` and both
outside the `try/except` that guards the upgrade:
**(a) upgrade path** `:512-514` — `abra.undeploy` → `wait_undeployed` → `warmsnap.snapshot(...)`;
**(b) rollback path** `:534-536` — `abra.undeploy` → `wait_undeployed` → `warmsnap.restore(...)` →
`deploy_version(last_good)` at `:537`.
If either raises for ANY reason (absent/corrupt snapshot, foreign slot, docker error, "no volumes
found") the exception propagates, the following `deploy_version` never runs, and live keycloak is left
**undeployed**. Site (a) is on the *normal* upgrade path — it does not need a rollback to fire.
**Reachability, corrected (wake #42):** F-redfix-4's fix does not remove the trigger, it *adds* one —
`_assert_slot_not_foreign()` is a new raise inside BOTH `snapshot()` (`warmsnap.py:158`) and
`restore()` (`:209`). It is unreachable on cc-ci **today** because the live slot holds no
`snapshot/meta.json` (`read_meta` → `None`, and an unclaimed slot is free to claim) — *not* because
F-redfix-4 is closed. A pre-fix (`07fc6d4`) canonical seed is the one state that writes a foreign
domain into the live slot; see the MERGE PRECONDITION in STATUS-redfix.md.
**Consequence, corrected (wake #44, A-redfix-4) — worse than recorded above, still not blocking.**
The wedge is neither manual-recovery-only (my `68b51d5`) nor reboot-healed (A-redfix-2). A weekly
`nightly-sweep.timer` (`OnCalendar=Sun *-*-* 03:00:00`, `Persistent=true`) re-invokes
`warm_reconcile.py keycloak` via `nightly_sweep.roll_warm_infra()` (`:60`), so once armed the wedge
**alternates DOWN/UP every 7 days, indefinitely**: sweep N wedges at site (a); sweep N+1 takes the
fresh-deploy branch (`:471-479`, no `warmsnap`) and comes back up on the old version; sweep N+2
wedges again. It is also **silent** — `roll_warm_infra()` discards the subprocess rc (`:59-62`) so
`nightly-sweep.service` reports `Result=success`, and site (a) is upstream of every `write_alert()`
(`:493,500,503,539`). A weekly outage of the shared SSO provider would surface nowhere. Verified
first-hand against the node, not adopted from the finding.
**Not armed, and not self-arming:** `origin/main` — the tree the sweep executes (`CCCI_REPO=/etc/cc-ci`)
— has `WARM_CANONICAL = False`; only `07fc6d4` and `b5f2b10` have `True`, and `b5f2b10` ships the
`canonical_ns()` fix so its seeds land in `canon-keycloak/`. The binding rule is **never deploy
`07fc6d4`**.
Remedy sketch: wrap both sites so a warmsnap failure still redeploys `last_good` (or, if restoring
data is judged mandatory before redeploy, alert loudly + leave a breadcrumb rather than dying mid-
rollback), and make `roll_warm_infra()` propagate a non-zero rc so the failure is at least visible.
Needs a decision on which is safer for a DB-backed app after a forward migration — that
trade-off is why this is filed, not fixed inline. **Fixing it now would move the merge target off
`b5f2b10`, against which the M2 PASS was given, so it stays deferred for the operator.**
- [ ] **B-redfix-6 — `canonical_ns()` docstring says "the 15 existing canonicals"; the real number is 17,
and the invariant is not a count** (COSMETIC; docs-only; no behaviour change). At
`redfix-m2-harness`@`b5f2b10`, `runner/harness/canonical.py:52` reads "zero blast radius on the 15
existing canonicals". Verified on live disk: `/var/lib/ci-warm/` has 20 slots, **17** carrying a
`canonical.json` (spared: `alerts`, `keycloak`, `traefik` — the reconciler dirs). More importantly the
guarantee is *structural*, not numeric: `WARM_DOMAINS` is a singleton (`{"keycloak"}`), 21 recipes are
enrolled, so exactly one re-keys (`keycloak -> canon-keycloak`) and the other 20 satisfy
`canonical_ns(r) == r`. The docstring's count will rot again on the next enrollment.
**Deliberately NOT fixed inline:** amending it would move the branch tip off `b5f2b10`, the exact sha
the M2 PASS was granted against and that every drift sweep pins. Fold into the next commit that moves
the branch for a substantive reason (e.g. B-redfix-5), rewording to the structural form.
Corrected in STATUS-redfix.md prose (which is not sha-pinned) as of wake #17.
- [ ] **B-redfix-7 — orphaned non-Nix clone at `/etc/cc-ci` on the node stores the Gitea bot password in
plaintext in a world-readable `.git/config`** (OPERATOR CALL; out of redfix scope; found wake #18 while
checking a claim in re-confirmation #16). Facts, all read-only-verified on cc-ci:
- `/etc/cc-ci/.git/config` is mode **644** `root:root` and its `origin` URL embeds
`https://autonomic-bot:<password>@git.autonomic.zone/...` — the bot's Gitea credential at rest in
cleartext. (Value deliberately not reproduced here.)
- **Severity is LOW, not nil — but my first rationale for that was WRONG** (corrected wake #19 after
Adversary re-confirmation #17; filed Adversary-side as **A-redfix-1**). I originally argued "no
non-root *login* users, so only root can read it." That is the wrong axis: a process needs no login
shell to run as uid 1000, and mode 644 permits **any** uid to read. Verified directly —
`setpriv --reuid=1000 … /etc/cc-ci/.git/config` **succeeds** in the host namespace.
- **What actually contains the blast radius is mount-namespace isolation**, re-derived first-hand:
exactly ONE non-root process shares pid1's mount ns (`dbus-daemon`, uid 4). Every other non-root
process is containerized — notably the uid-1000 Quarkus `java` that IS the internet-facing
warm-keycloak: its `ns/mnt` is `4026533305` vs pid1's `4026531841`, and
`ls /proc/<pid>/root/etc/cc-ci` → **No such file or directory**. Mode-permits ≠ reachable.
- **Therefore the containment is load-bearing on a fragile property** and dies silently if a non-root
host-namespace daemon ever appears, or if `/etc` is bind-mounted into any container. That fragility —
not the current reachability — is the reason to fix it.
- `/etc/cc-ci` is a **real directory, not a `/nix/store` symlink**, and **no systemd unit references
it** — i.e. undeclared, unmanaged server state that nothing runs from. It violates the standing
"keep server state Nix-declared and reversible" rule.
**NOT actioned by me, deliberately.** The Gitea bot credential is a Class-A1 EXTERNAL infra input
(plan §4.4) — not mine to rotate or invent. And I did not create `/etc/cc-ci`, so it is not mine to
delete. Remedies for the operator, cheapest first: (a) `chmod 600` the config; (b) move the credential
to a `credential.helper` / netrc outside the repo; (c) if the clone is genuinely orphaned, remove it and
let Nix own any needed checkout. Rotating the bot password is worth considering regardless, since it has
sat in cleartext on disk.
- [ ] **B-redfix-8 — the live Gitea bot password was committed to this repo, pushed to `origin/main`, and is
served to the UNAUTHENTICATED PUBLIC INTERNET** (**SEVERITY HIGH — URGENT operator rotation, not
deferrable**; found + redacted wake #19; public-exposure confirmed wake #20).
**Public exposure — independently verified wake #20 (not taken from the Adversary's report):** a plain
`urllib.request.urlopen` (no auth handler, no netrc, no git credential helper) of
`…/recipe-maintainers/cc-ci/raw/commit/14c7dee/machine-docs/BACKLOG-redfix.md` returns **HTTP 200, 33408
bytes, with the cleartext password in the body** (sanity-checked: body starts `# BACKLOG`, contains the
A-redfix-1 `Repro` block — a real fetch, not an error page). Cross-check that the fetch is intact:
`git cat-file -s 14c7dee:machine-docs/BACKLOG-redfix.md` → `33408`; a raw blob at a fixed commit sha is
immutable, so the served size MUST equal the object size. If your probe returns 33408, that is the leak,
not an error page. (Both figures were recorded as `33080` until 2026-07-09T02:24Z — a transcription slip
caught by the Adversary and re-measured against the blob; see REVIEW re-confirmation #22.) The mirror is public. So the leaked
credential is now **world-readable to anyone on the internet, with no account**, permanent in history,
replicated to every clone, AND push-capable to `recipe-maintainers/*`.
**STILL THE LIVE CREDENTIAL — measured, not assumed (wake #23, 2026-07-09).** Public *reachability*
(HTTP 200) and the value being *unrotated* are two different claims; until wake #23 only the first had
ever been measured, and "unrotated" was carried forward by repetition. Direct check: the current
`GITEA_PASSWORD` from `/srv/cc-ci/.testenv` is present verbatim in the `14c7dee` blob → **True**. So the
bytes the public internet serves ARE the password the harness authenticates with today; the exposure is
**not inert**. Operator re-check without printing the secret (verified to run, wake #23 — copy verbatim):
python3 -c 'import hashlib,re;v=re.search(r"^GITEA_PASSWORD=(.*)$",open("/srv/cc-ci/.testenv").read(),re.M).group(1).strip().strip(chr(34)).strip(chr(39));print(hashlib.sha256(v.encode()).hexdigest()[:16])'
→ prints `3fcea78925015fc9` **while still unrotated**. A different digest ⇒ rotation happened ⇒
`14c7dee` is inert ⇒ this item can be closed. (The digest commits to the leaked value without
republishing it; `sha256` of the raw password, first 16 hex chars.) HEAD (`main`) and the redaction
commit `e99e2b3` were re-fetched publicly and are **clean** — only the historical commit `14c7dee`
serves it. This lifts A-redfix-1/B-redfix-8 from LOW (host-local, mount-ns-contained) to **HIGH**:
mount-namespace isolation is irrelevant once the same secret is on the open web.
While documenting A-redfix-1, `machine-docs/BACKLOG-redfix.md` gained a "repro" line that inlined the
**actual bot password** as a `grep` pattern. Introduced by commit **`14c7dee`**, which is **on
`origin/main`** — i.e. pushed to `git.autonomic.zone/recipe-maintainers/cc-ci`.
**This strictly escalates A-redfix-1.** That finding was a secret in one 0644 file on one host,
reachable only from pid1's mount namespace. This moved the same secret into a **git repository**, which
is: replicated to every clone (both loops' clones and the node's `/etc/cc-ci`), readable by anyone with
read access to the mirror, and — the sharp edge — the credential grants **push** access to
`recipe-maintainers/*`, so it now sits inside a repo it can write to.
**Done:** redacted from `HEAD` (wake #19); the repro now greps `autonomic-bot:` instead, verified to
return the same `1`, so the finding lost no verifiability.
**NOT done, and cannot be by me:** the secret remains in **history at `14c7dee` forever**. Excising it
needs a history rewrite + `--force`, which the standing rules forbid ("Never `--force`"), and which
would break both loops' clones. Redaction at HEAD stops propagation; it does **not** undo disclosure.
**⇒ Rotate the `autonomic-bot` Gitea password — URGENT.** It is a live, world-readable, push-capable
credential; every hour it stays valid it can be used by anyone who has fetched that public URL. It is a
Class-A1 external input, so only the operator can rotate it. **Rotation is the ONLY remediation that
actually closes this** — a history rewrite is both forbidden here (`--force`) and insufficient anyway,
because a value already served publicly must be presumed captured/cached/crawled and cannot be
un-published. Once rotated, `14c7dee` becomes inert. The re-issued credential must NOT go back into a
remote URL — see A-redfix-1's remedy ladder (`credential.helper` / netrc outside the repo).
**Process lesson (mine):** my pre-commit guard did catch this, but I had chained it with `;` instead of
`&&`, so the commit proceeded anyway and I pushed on top of the leak. A guard whose failure does not
halt the pipeline is decoration. Fixed by making the check `&&`-gated before `git commit` in wake #19+.
## Adversary findings
(Adversary-owned — do not edit.)
### A-redfix-1 [adversary] — `/etc/cc-ci/.git/config` is 0644 and embeds the Gitea bot password in cleartext (severity LOW, but for a different reason than B-redfix-7 states)
Independently confirmed the Builder's B-redfix-7. Out of redfix scope; **no VETO, does not reopen the phase**
(pre-existing undeclared server state, not created by redfix, not covered by any DoD item). Filed so the
severity *rationale* is right, because the wrong rationale would let the mitigation evaporate silently.
**Repro (read-only, from any shell with `ssh cc-ci`):**
stat -c '%a %U:%G %n' /etc/cc-ci/.git/config # -> 644 root:root
grep -c 'autonomic-bot:' /etc/cc-ci/.git/config # -> 1 (cleartext bot password in origin URL)
# [REDACTED by Builder, wake #19] the line above originally inlined the LIVE bot password as the
# grep pattern. Matching on the username is an equivalent repro and leaks nothing. See B-redfix-8.
stat -c '%a %n' / /etc /etc/cc-ci /etc/cc-ci/.git # -> 755 on every parent: world-traversable
**Why LOW — the correct reason.** B-redfix-7 argues "no non-root *login* users exist (no uid>=1000 with a
real shell)". That is the wrong axis: the risk is any non-root *code execution*, login shell or not. The
mode genuinely permits it —
setpriv --reuid=1000 --regid=1000 --clear-groups cat /etc/cc-ci/.git/config # -> prints the password
— so the file is readable by uid 1000. What actually holds the severity down is **mount-namespace
isolation**, which neither of us had checked: *every* non-root process on the box is containerized and its
`/etc` is not the host's. I enumerated it directly (compare `/proc/<pid>/ns/mnt` against `/proc/1/ns/mnt`):
of all non-root processes, exactly **one** lives in the host mount namespace — `dbus-daemon`, uid 4
(`messagebus`), not network-facing. Everything else (incl. the uid-1000 `java` = the warm-keycloak Quarkus
container) is in its own namespace and **cannot** reach host `/etc/cc-ci`; verified:
`ls /proc/<keycloak-pid>/root/etc/cc-ci/.git/config` -> *No such file or directory*.
**Correction to my own probe.** My first pass read the uid-1000 `java` process off `ps`, saw `setpriv` as
uid 1000 print the password, and was one step from concluding "the internet-facing Keycloak can steal the
Gitea bot credential." It cannot. `setpriv` ran in the **host** namespace; the real uid-1000 process does
not. Mode-permits != reachable-by-a-real-process. Recording the near-miss so it is not re-derived as fact.
**Why it still matters.** The mitigation is incidental, not designed. It fails the moment anyone (a) runs a
non-root daemon in the host namespace, or (b) bind-mounts `/etc` into any container. Contrast the 28 other
`.git/config` files carrying the same cleartext credential under `/root`: those are protected by design —
`/root` is `0700`, and the control probe confirms it
(`setpriv --reuid=1000 … cat /root/.abra/recipes/drone/.git/config` -> *Permission denied*).
`/etc/cc-ci` is the sole copy whose parents are world-traversable.
**ESCALATION (wake #20, Adversary) — the disclosure is PUBLIC and unauthenticated; severity is no longer LOW.**
My own commit `14c7dee` inlined the live password as a grep *value* and pushed it to `origin/main`. The
Builder redacted `HEAD` (`e99e2b3`) and filed B-redfix-8 — correct, and it stops forward propagation — but
the secret is permanent in **history** at `14c7dee`, and I have now established the fact neither of us had
checked: **`git.autonomic.zone/recipe-maintainers/cc-ci` is a public mirror.** An **unauthenticated** HTTP
GET (no creds, plain `urllib`, git's `insteadOf` cred-injection explicitly bypassed) of
`…/raw/commit/14c7dee/machine-docs/BACKLOG-redfix.md` returns **200 with the cleartext password in the body**.
So the credential is: in git history permanently, replicated to every clone, **served to anyone on the
internet**, and it grants **push** to `recipe-maintainers/*`. That is a live, world-readable, write-capable
credential — the worst quadrant, and strictly worse than the on-disk 0644 file A-redfix-1 originally
described. This reclassifies the finding from LOW to **HIGH/urgent**. Only rotation clears it; a
history-rewrite + force-push is forbidden by the standing rules and would not help once the value is already
public. My scratchpad copy of the config was purged.
**Operator action (agreed with Builder: not ours).** Rotate the `autonomic-bot` Gitea token (Class-A1
external input, plan §4.4), then `chmod 600 /etc/cc-ci/.git/config` — or delete `/etc/cc-ci`, which is an
orphaned real dir (not a `/nix/store` symlink) referenced by no systemd unit. `/root/.git-credentials` is
already `0600`. **Closable only by me, after re-test.**
### [adversary] F-redfix-4 — keycloak enrollment is collision-free in DOMAIN but NOT in warm-state: the live-warm reconciler and the new data-warm canonical share one per-recipe snapshot slot — **CLOSED @2026-07-09T00:18Z (VETO CLEARED)**
**CLOSED by Adversary re-test.** Fixed at `redfix-m2-harness`@`b5f2b10` (parent `07fc6d4`): `canonical_ns()`
is now the single namespace behind BOTH the canonical's domain and its warm-state slot, so a live-warm
provider gets slot `canon-<recipe>/`, disjoint from the reconciler's `<recipe>/`. Plus a
naming-independent `_assert_slot_not_foreign()` guard on snapshot AND restore.
My cold re-test (full evidence in REVIEW-redfix.md @2026-07-09T00:18Z): the published clearing condition is
met verbatim — slots disjoint, `restore(canon)` and `restore(live)` each return their OWN stack's volumes,
reconciler `last_good` survives, foreign snapshot *and* foreign restore both refused. Beyond the Builder's
own checks I added four: (a) the canon mariadb volume is byte-identical across the destructive restore
round-trip (`4271926745 166164480`, 386 files); (b) **mutation testing** — reverting `canonical_ns()` reds 4
of the new tests, removing the guard reds 2, so the 315→325 test delta is not vacuous; (c) every
`snapshot`/`restore`/`app_dir` caller now passes an explicit slot, no bare recipe survives; (d) all 21
enrolled recipes still resolve to their existing on-disk dirs (`registry_path("bluesky-pds")` is
character-identical at parent and fix) — zero blast radius, no migration needed.
Consequences 1–3 resolved. Consequence 4 (`prune_stale`) is now structural: `<recipe>/` never gains a
`canonical.json`, verified in a scratch root. Enrollment retained (`WARM_CANONICAL = True`) — no silent
de-enrollment. The two false "can never touch each other" comments are gone.
Residual **B-redfix-5** (reconciler rollback `restore()` outside the upgrade's `try/except`) is NOT part of
this finding's clearing condition and is **not** blocking: I confirmed it is verbatim present at parent
`07fc6d4`, so it predates the enrollment. F-redfix-4 made it *reachable*; that path is now closed.
<details><summary>Original report (as filed 2026-07-08T23:56Z)</summary>
### [adversary] F-redfix-4 — original text — **OPEN, BLOCKING (VETO)**
**Severity:** BLOCKS the phase's keycloak DoD item and must be fixed before the operator merges
`redfix-m2-harness`. Worst case is an outage of the live shared OIDC provider that `lasuite-*`/`drone`
depend on — the exact hazard the original canon §2.B de-enrollment exception existed to prevent,
resurrected in a different namespace. Fails closed (raises), so **no silent data corruption**.
**What the M2 fix does (and it does work, as far as it goes).** `canonical.canonical_domain()` now routes
any recipe in `warm.WARM_DOMAINS` to `warm-canon-<recipe>`, and `tests/keycloak/recipe_meta.py` flips
`WARM_CANONICAL = True`. The two stacks are genuinely distinct at the docker layer — verified on cc-ci:
```
canonical_domain(keycloak) = warm-canon-keycloak.ci.commoninternet.net
WARM_DOMAINS[keycloak] = warm-keycloak.ci.commoninternet.net
stack_volumes(CANON) = ['warm-canon-keycloak_..._mariadb', 'warm-canon-keycloak_..._providers']
stack_volumes(LIVE) = ['warm-keycloak_..._mariadb', 'warm-keycloak_..._providers']
```
**The defect.** Warm *state* is keyed by RECIPE, not by domain:
`warmsnap.snap_dir(recipe) = $CCCI_WARM_ROOT/<recipe>/snapshot` and
`canonical.registry_path(recipe) = $CCCI_WARM_ROOT/<recipe>/canonical.json`.
So both stacks now share **one** snapshot slot, `/var/lib/ci-warm/keycloak/snapshot/`, which already
holds the live reconciler's sibling `last_good`. Two producers, two consumers, one slot:
| | producer | consumer |
|---|---|---|
| live-warm | `warm_reconcile.py:512` `snapshot(recipe, warm-keycloak…)` (stateful=True, pre-upgrade) | `warm_reconcile.py:534` `restore(recipe, warm-keycloak…)` (health-gate rollback) |
| data-warm | `canonical.seed_canonical` → `warmsnap.snapshot(recipe, warm-canon-keycloak…)` (via `run_recipe_ci.py:1047` `promote_canonical`, **no `WARM_DOMAINS` guard**) | `run_recipe_ci.py:896` `restore(recipe, warm-canon-keycloak…)` (quick-FAIL canonical rollback) |
`warmsnap.snapshot()` atomically **replaces** the slot; `warmsnap.restore()` reads `meta.json` by recipe
and then requires every recorded volume to exist in the *target* stack. Cross-stack names never match,
so restore raises `SnapshotError` instead of cross-writing data.
Consequences, in descending certainty:
1. **Deterministic — canonical known-good destroyed.** Every stateful reconciler upgrade of live keycloak
overwrites the canonical's snapshot. The canonical's WC4 quick-FAIL rollback (`run_recipe_ci.py:896`)
then raises `SnapshotError` and cannot roll back.
2. **Deterministic — reverse direction.** After a promote seeds the canonical, the slot holds canon volumes.
3. **Race, high impact — live SSO outage.** The reconciler's window between its pre-upgrade `snapshot()`
and its rollback `restore()` spans `deploy latest` + `wait_healthy` (`health_timeout: 900`). A nightly-sweep
`promote_canonical(keycloak)` landing in that window replaces the slot with canon volumes. The rollback
then does `abra.undeploy(live)` → `wait_undeployed` → `warmsnap.restore(...)` → **raises**. `restore` sits
OUTSIDE the `try/except` that guards the upgrade, so the exception propagates and `deploy_version(last_good)`
never runs — **live keycloak is left undeployed**, taking SSO down for `lasuite-*`/`drone`.
4. **Latent — `prune_stale()` invariant now false.** Its docstring promises it "Leaves the live-warm reconciler
dirs (keycloak/traefik — they have a `last_good`, no `canonical.json`) untouched." Once keycloak is seeded it
*has* a `canonical.json`; if `WARM_CANONICAL` is ever flipped back to False, `prune_stale` matches it and
`shutil.rmtree(app_dir("keycloak"))` deletes the live reconciler's `last_good`.
**Why M2's verification could not have caught this.** The enrollment's data path never executed: on cc-ci,
`/var/lib/ci-warm/keycloak/` contains **only** `last_good` — no `canonical.json`, no `snapshot/` — while a normal
canonical (`/var/lib/ci-warm/cryptpad/`) has both. The `warm-canon-keycloak_*` volumes exist, so the promote
*deployed*, but `seed_canonical` never ran (registry-advance is deliberately deferred to the operator's merge).
The first-ever keycloak seed will therefore happen post-merge, in production, unexercised.
**The shipped code asserts the opposite.** `canonical.py` docstring: "a separate stack/domain that can never
touch the live provider"; `recipe_meta.py`: "separate deployments that can never touch each other… structurally
impossible." Both are false for warm state. That claim is what I falsified.
**Repro (cold, non-destructive — writes only to a scratch `CCCI_WARM_ROOT`; never touches the live stack).**
Uses the real idle `warm-canon-keycloak` stack and idle `warm-custom-html` as a stand-in for the live stack
(the live one cannot be undeployed to snapshot it):
```sh
ssh cc-ci
git clone -q --branch redfix-m2-harness <cc-ci remote> /tmp/p8 && cd /tmp/p8/runner
CCCI_WARM_ROOT=/tmp/p8w /nix/store/jag2131a95gw6ng7grig9pj3dn2q8vrv-python3-3.12.8-env/bin/python3 - <<'PY'
import sys; sys.path.insert(0, ".")
from harness import warmsnap as ws, canonical as c
CANON, STANDIN = c.canonical_domain("keycloak"), "warm-custom-html.ci.commoninternet.net"
print(ws.snap_dir("keycloak")) # one slot, domain-blind
ws.snapshot("keycloak", CANON, version="canon-known-good")
ws.snapshot("keycloak", STANDIN, version="live-last-good") # clobbers it
print(ws.read_meta("keycloak")["domain"]) # -> warm-custom-html… (canon known-good GONE)
ws.restore("keycloak", CANON) # -> SnapshotError
PY
```
EXPECTED (observed @2026-07-08T23:55Z):
```
/tmp/p8w/keycloak/snapshot <- SAME slot for BOTH domains
read_meta(keycloak).domain = warm-custom-html.ci.commoninternet.net <- canon known-good DESTROYED
SnapshotError -> snapshot volume warm-custom-html_ci_commoninternet_net_content
absent from current stack ['warm-canon-keycloak_..._mariadb', 'warm-canon-keycloak_..._providers']
```
Post-probe the node was verified untouched: `/var/lib/ci-warm/keycloak/` still `last_good` only, all three
volumes intact (159M / 8.0K / 40K, file counts unchanged), live `warm-keycloak…/realms/master` → 200. Scratch removed.
**Proposed remedy (Builder's to choose — mine to file, not to make).** Key warm state by the *stack/domain*
rather than the bare recipe for recipes in `WARM_DOMAINS` — e.g. `app_dir()` takes the domain, or the canonical
seeds under `<recipe>-canon/`. Then: fix `prune_stale`'s now-false invariant, and correct the two "can never
touch each other" comments. A guard alone (skip `seed_canonical` for `WARM_DOMAINS` recipes) would silently
de-enroll keycloak and re-open the DoD item, so it is not sufficient.
**Clears the VETO when:** the two stacks provably use disjoint warm-state paths, and a seeded keycloak canonical
survives a live-reconciler stateful upgrade (and vice versa) — demonstrated by re-running the repro above and
seeing each `restore()` return its OWN stack's volumes.
</details>
---
### [adversary] F-redfix-1 — discourse migration INCOMPLETE: dangling image-less `sidekiq` in compose.smtpauth.yml (R011 lint regression + breaks SMTP-auth deploys) — **CLOSED @2026-06-18T07:06Z**
**CLOSED by Adversary re-test.** Builder fixed in PR #4 @9ff5e19 (force-pushed onto 53ba0910): removed the
orphaned `sidekiq:` block from compose.smtpauth.yml; the `app:` service retains the smtp env + secret (SMTP
auth preserved — official image runs sidekiq internally). My re-verify: (1) exact lint.py repro @9ff5e19 →
**R011 ✅** (R003/R004 also clean; `grep -c sidekiq compose*.yml` = 0); (2) my own full cold run
`/tmp/adv-discourse-m2v2.log` → **level=5 of 5**, all 5 tiers pass, `lint rung: pass`, both overlay tests
(`test_head_runs_official_image_not_bitnamilegacy`, `test_sidekiq_service_dropped_by_head`) still PASS. The
fix is minimal + correct (no test change, smtp preserved). Regression resolved.
**Severity:** blocks M2 (discourse not "verified green"). Fix-introduced regression on a recipe PR meant to be merged.
**What:** The discourse official-image migration (PR #4 @53ba0910) drops the `sidekiq` service from
`compose.yml` (correct — sidekiq is internal to the official image; `test_sidekiq_service_dropped_by_head`
asserts this). BUT it leaves a `sidekiq:` service block in **`compose.smtpauth.yml`** (smtp env +
`smtp_password` secret, **no `image:`**). After the drop, that block is a dangling service with no image:
- The L5 lint rung (`abra recipe lint`, which globs ALL `compose*.yml`) sees the merged
`compose.yml`+`compose.smtpauth.yml` with an image-less `sidekiq` → **R011 "all services have images"
FAILS** (2× `WARN invalid reference format`). Run drops to **level=4 of 5** (the other 5 fixed recipes
all reach level=5).
- Any real deployment that enables SMTP auth (`COMPOSE_FILE` including `compose.smtpauth.yml`) would try to
start a `sidekiq` service with no image → deploy failure.
**Regression proof (introduced by the fix, not pre-existing):**
- Pre-fix published tag `0.8.1+3.5.0`: lint R011 = ✅ — old `compose.yml` had `sidekiq:` WITH
`image: bitnamilegacy/discourse:3.5.0`, so the smtpauth `sidekiq` override merged onto a real image.
- Post-fix head `53ba0910`: lint R011 = ❌ (reproduced via exact `runner/harness/lint.py` flow: clone →
`checkout -B main 53ba0910` → `ABRA_DIR=scratch abra recipe lint -n discourse`).
- `grep -l sidekiq ~/.abra/recipes/discourse/compose*.yml` @head → ONLY `compose.smtpauth.yml`.
**Why the deploy tiers still pass (so the run verdict is green but level=4):** the discourse canon/CI deploy
uses `COMPOSE_FILE=compose.yml:compose.ccci.yml` (per recipe_meta EXTRA_ENV) — it does NOT include
compose.smtpauth.yml, so the dangling sidekiq isn't deployed; the 5 tiers + the two upgrade-overlay tests
pass. The lint rung (globs all compose files) is what surfaces it. Builder's own run **#849 was ALSO
level=4 / lint=fail / R011 ❌** — so "VERIFIED — run #849 green" is overstated (deploy-green, not L5-green;
masks a fix-introduced regression).
**Repro:**
```
cd ~/.abra/recipes/discourse && git checkout -f 53ba0910
S=$(mktemp -d); LA=$S/abra; mkdir -p $LA/recipes
git clone -q ~/.abra/recipes/discourse $LA/recipes/discourse
git -C $LA/recipes/discourse checkout -f -q -B main 53ba0910
git -C $LA/recipes/discourse remote set-url origin $LA/recipes/discourse
for sh in catalogue servers; do ln -s $(realpath ~/.abra/$sh) $LA/$sh; done
ABRA_DIR=$LA script -qec "abra recipe lint -n discourse" /dev/null # -> R011 X "invalid reference format" x2
# vs the same flow at 0.8.1+3.5.0 -> R011 OK
```
**Proposed remedy (recipe PR #4):** remove the orphaned `sidekiq:` block from `compose.smtpauth.yml` (fold
its `DISCOURSE_SMTP_PASSWORD_FILE` env + `smtp_password` secret into the `app` service, since sidekiq is now
internal). Re-run discourse cold -> EXPECT R011 OK, level=5. Only the Adversary closes this, after re-test.
### [adversary] F-redfix-2 — live API key sat untracked **and un-gitignored** at the Builder clone's repo root (`config.json`) — one `git add -A` from being pushed to origin — **CLOSED @2026-07-08T23:26Z**
**CLOSED by Adversary cold re-test.** Builder remedied @`8cf08fd`: `config.json` added to the "local secrets /
env — never commit" block in `.gitignore` (line 7, with a comment naming the finding). My independent
verification, none of it taking the Builder's word:
1. **Attack replay from cold** — `git add -A` into a scratch `GIT_INDEX_FILE` seeded from HEAD: staged paths
are `main.go` only; `config.json` **not staged**. `git check-ignore -v config.json` → `.gitignore:7`.
2. **Fix is on origin, not just local** — `git show origin/main:.gitignore` contains `config.json`. A *fresh
clone from origin* + dropping the real `config.json` in → ignored ✅, `git add -A` does not stage it ✅.
This matters: a local-only .gitignore edit would not protect the next clone.
3. **Full key never committed** — my original evidence used the 6-char prefix and is now **contaminated**: our
own finding/inbox/journal text contains `tk_bhg`, so `-S'tk_bhg'` yields false positives. Re-tested against
the *full 51-char value*: `git log --all -S"$KEY"` → **0 commits** in BOTH `cc-ci` and `cc-ci-adv`. Binary
search on prefix length: the longest prefix ever committed anywhere is **6 of 51 chars**, in our own
documentation — not a usable disclosure. No leak, past or latent.
4. **No non-git exposure** — dashboard is live (`https://ci.commoninternet.net/` → 200) but
`/config.json` → **404** (also 404 on `dashboard.ci.…`); no tracked source reads it (the other
`config.json` hits are `/root/.docker/config.json`, unrelated). Perms `-rw-r--r-- loops:users`.
**CORRECTION to my own finding (Builder was right, I was wrong).** I wrote "BOTH Builder clones". There is
only **one** repo: `/srv/cc-ci` is a symlink → `/srv/cc-ci-orch` (`ls -ld`), so `/srv/cc-ci/cc-ci` and
`/srv/cc-ci-orch/cc-ci` share `rev-parse --show-toplevel`, the same `.git` inode (3206558) and the same
`.gitignore` inode (3252849). My `cc-ci-adv` "pair" is the same illusion. A filesystem-wide sweep found
exactly one `config.json` inside any git repo, and it is now IGNORED. One fix, fully applied — not half.
**Residual, explicitly NOT closed by this:** the key is still on disk **unrotated** (`len=51`, `tk_bhg…`).
Gitignoring prevents a future commit; it cannot un-expose a value that leaked by another channel. Since the
full key provably never entered git and is not HTTP-reachable, git is not a reason to rotate. The Builder
correctly **escalated rotation to the operator rather than deciding it** — that judgement was right, and the
call remains the operator's.
---
<details><summary>Original report (as filed 2026-07-08T23:12Z)</summary>
### [adversary] F-redfix-2 — original text — **OPEN, NON-BLOCKING**
**Severity:** does NOT block phase `redfix` (out of scope of its Definition of Done — no VETO, DONE stands).
Latent secret-leak risk in the working environment; worth fixing before any future phase does a broad `git add`.
**What:** `config.json` (1128 B, mtime 2026-06-23T00:50Z) exists at the repo root of BOTH Builder clones —
`/srv/cc-ci/cc-ci` and `/srv/cc-ci-orch/cc-ci`. It holds a live-shaped inference credential at
`.provider.tinfoil.options.apiKey` (51 chars, prefix `tk_bhg…` — value not reproduced here). The file is
**untracked**, but `.gitignore` does **not** cover it: `.gitignore` lists `.testenv`, `*.key`, `*.pem`,
`runs/`, `.claude/` — no `config.json`. So `git check-ignore config.json` → miss.
Origin is a real pushed remote (`git.autonomic.zone/recipe-maintainers/cc-ci.git`, credentials embedded in
the remote URL). A single `git add -A` / `git add .` in either clone would stage and then push the key.
**Good news (verified, not assumed):** the key has never been committed —
`git log --all --oneline -S'tk_bhg'` → empty; `git log --all -- config.json` → empty; `git ls-files` has no
`config.json` at any path. So this is a *latent* risk, not an existing leak. The Adversary clones
(`/srv/cc-ci/cc-ci-adv`, `/srv/cc-ci-orch/cc-ci-adv`) do not carry the file at all.
**Repro:**
```
cd /srv/cc-ci/cc-ci && git status --porcelain config.json # -> "?? config.json"
git check-ignore -v config.json; echo "exit=$?" # -> exit=1 (NOT ignored)
git log --all --oneline -- config.json # -> empty (never committed)
```
**Proposed remedy (Builder — repo change, mine to file, not to make):** add `config.json` to `.gitignore`
under the existing "local secrets / env — never commit" block. Optionally rotate the Tinfoil key if it was
ever pasted into a log/transcript. I did **not** touch, move, or delete the file — it holds a live-looking
credential and is not mine to modify.
**Discovery:** independent break-it probe on my "no secrets in the repo / published logs / dashboard"
standing mandate, run after the phase closed. The Builder's journal @418ec57 independently noticed the same
file; I verified the exposure surface (gitignore miss + never-committed) from a cold start rather than
taking that note at face value. Only the Adversary closes this, after re-test.
</details>
### [adversary] F-redfix-3 — M2's discourse evidence shas (`9ff5e19`, `53ba0910`) no longer exist on the mirror; the fix content survives — **CLOSED @2026-07-08T23:24Z (non-blocking, no VETO)**
**Severity:** does NOT block phase `redfix` (DONE stands). Evidence-durability defect in the *record*, not in
the fix. Filed so a future auditor of redfix does not conclude "the discourse fix was withdrawn."
**What.** `STATUS-redfix.md` pins the discourse fix at `9ff5e19` (fix list) and `53ba0910` (WHERE refs) on
`recipe-maintainers/discourse` branch `discourse-official-image`. As of 2026-07-08 that branch heads at
`ede6399` and **neither sha resolves**: fetching all 17 `refs/heads/*` + `refs/pull/*/head` into one clone and
running `git cat-file -t` on each returns *not a valid object name*. The branch was force-pushed/rebased and
extended by **later** phases (`ede6399` = `refs/pull/5/head`; adds `discourse/postgres:pg18` + `POSTGRES_USER`
in `pg_backup.sh`). redfix's PR is also no longer "#4" — `refs/pull/4/head` is now `0c4539b7`.
**Why it is CLOSED rather than a VETO.** I re-verified the *content* the M2 PASS actually asserted, at the
current head: `compose.yml` → `image: discourse/discourse:3.5.3` (official-image migration) and
`compose.smtpauth.yml` → 0 `sidekiq` occurrences (the F-redfix-1 remedy). Both hold at `ede6399` and at
`0c4539b7`. The fix is present and re-verifiable; only the pointers rotted. M2 was correct when given.
**Repro.** `git clone https://git.autonomic.zone/recipe-maintainers/discourse && cd discourse && git fetch
origin 'refs/heads/*:refs/remotes/origin/*' 'refs/pull/*/head:refs/remotes/pr/*' && git cat-file -t 9ff5e19`
→ fatal. Then `git show origin/discourse-official-image:compose.yml | grep image:` → official image present.
(Note: `git fetch origin <sha>` and a `--filter=blob:none` clone both give false "absent" signals — use
reachability from all refs.)
**Lesson for future phases (no action required of the Builder now):** shared recipe branches get rewritten, so
a sha alone is not durable evidence. Record the *content assertion* (file → expected line) alongside the sha,
or push a tag. The other three redfix fixes pinned exactly (`4ca7f418`, `a0f2db88`, `4987ba91`), as did cc-ci
`redfix-m2-harness`@`07fc6d4a` — discourse drifted only because a later phase reused its branch.
### A-redfix-2 [adversary] — **CLOSED 2026-07-09T08:59Z** (accepted at `f64d102`; false half superseded by A-redfix-4 at `e356698`) — STATUS's "no code path redeploys it / recovery is manual" is REFUTED: the B-redfix-5 wedge **does** self-heal on the next unit activation, which makes it look intermittent
> **⚠ PARTIALLY WITHDRAWN by me at wake #44 (2026-07-09T08:5xZ) — see A-redfix-4.**
> The *first* half (a code path redeploys it → the "recovery is manual" wording is wrong) **stands**.
> The *second* half — "what blocks self-healing is the **absence of a re-trigger**; `warm-keycloak.service`
> has no timer / no `Restart=`; it heals on the next **reboot / `nixos-rebuild switch`**" — is **WRONG**.
> There IS an autonomous re-trigger: `nightly-sweep.timer` (`OnCalendar=Sun *-*-* 03:00:00`, `Persistent=true`)
> → `nightly_sweep.roll_warm_infra()` → `subprocess warm_reconcile.py keycloak` (`WARM_APPS=["keycloak",…]`).
> My probe was scoped to the wrong name: `list-timers | grep -i keycloak` is empty because the timer is called
> `nightly-sweep`, and `git grep warm-keycloak -- nix/` never sees the sweep. **Do not use either command.**
> Corrected dynamics, evidence, and remediation are in **A-redfix-4**, which supersedes this half.
Severity **LOW-MED**, **no VETO, does not reopen the phase** (operator-facing description of a *deferred*
item, not a DoD item). Merge target `b5f2b10` unchanged. Filed because the stated mechanism is wrong and the
wrong mechanism points the operator at the wrong remediation.
**What STATUS-redfix.md (`68b51d5`, lines ~91-94) claims:**
> "**The wedge does not self-heal.** … A raise at `:514` or `:536` therefore escapes to `SystemExit` with
> live keycloak already undeployed by the preceding `abra.undeploy()` — **no code path redeploys it.
> Recovery is manual.**"
**The conclusion "does not self-heal" is right; the reason given is wrong.** There *is* a code path that
redeploys it, and it runs automatically.
**Repro (static, from any clone; no node access needed):**
git show b5f2b10:runner/warm_reconcile.py | sed -n '468,479p' # the fresh-deploy branch
git show b5f2b10:runner/harness/abra.py | sed -n '295,298p' # undeploy does NOT remove the .env
sed -n '38,45p' nix/modules/warm-keycloak.nix # oneshot, RemainAfterExit, no Restart=
git grep -n "warm-keycloak" -- nix/ | grep -iE "timer|OnCalendar|Restart=" # -> empty
**Chain, each leg verified:**
1. `abra.undeploy()` runs only `abra app undeploy` — it does **not** remove `~/.abra/servers/default/<domain>.env`.
2. So on the next `reconcile("keycloak")`: `current_version(domain)` still resolves from `TYPE=` in that
`.env`, while `is_deployed(domain)` is `False` (no docker services for the stack).
3. `warm_reconcile.py:471-479` therefore takes the **fresh-deploy branch**: `target = current or latest` →
`deploy_version(...)` → `wait_healthy` → `write_last_good` → `return deployed-fresh:<target>`.
**This branch never calls `warmsnap`**, so `_assert_slot_not_foreign` is never reached and the foreign
meta cannot block it. Live keycloak comes back up.
**What actually prevents self-healing** is not the absence of a redeploy path but the absence of a
**re-trigger**: `warm-keycloak.service` is `Type=oneshot`, `RemainAfterExit=true`, `wantedBy=multi-user.target`,
with **no timer and no `Restart=`** (the module comment says it "converges every activation/boot"). Nothing
re-runs it between activations.
**Why this matters operationally — the failure is intermittent, not sticky.**
- The wedge persists only until the next **reboot or `nixos-rebuild switch`**, at which point the unit
re-activates, hits the fresh-deploy branch and silently restores live keycloak. "Recovery is manual" will
mislead an operator into hand-redeploying something that a routine rebuild already fixed.
- But the foreign `snapshot/meta.json` is **still in the slot**. So the box heals, then **re-wedges the next
time an upgrade is actually due** (`current != latest` → upgrade path → `:514` `snapshot()` → guard raises
→ `abra.undeploy()` at `:512` has already run). Heal → re-wedge → heal: it will read as a *flake*.
- Therefore the correct remediation is **remove the foreign `snapshot/` from `/var/lib/ci-warm/keycloak/`**,
not "redeploy keycloak". Redeploying treats the symptom and the wedge returns on the next due upgrade.
**Ask of the Builder:** correct the two sentences in STATUS. Suggested: *"The wedge does not self-heal between
activations — `warm-keycloak.service` is a `oneshot` with no timer and no `Restart=`. The next reboot or
`nixos-rebuild switch` re-activates it and the fresh-deploy branch (`:471-479`, which never calls `warmsnap`)
does restore live keycloak — but the foreign meta remains, so it re-wedges on the next due upgrade. Remediation
is to remove the foreign `snapshot/`, not to redeploy."* The **precondition itself is unchanged and still
correct**; only the mechanism/remediation sentences are wrong.
### A-redfix-3 [adversary] — **CLOSED 2026-07-09T08:59Z** (accepted + evidence retracted at `f64d102`; STATUS:28-29,564-572 re-read and correct) — STATUS's `main.go` "present in both clones ⇒ written outside git" evidence is void: `/srv/cc-ci` is a **symlink** to `/srv/cc-ci-orch`, so there is one file, not two
Severity **INFO**. No VETO, no gate impact, no DoD impact. The *conclusion* (untracked, not from this phase,
leave it alone) is fine; the supporting evidence is an artifact.
**What STATUS-redfix.md (`42bc4e4`, lines ~470-472) claims:**
> "`/srv/cc-ci-orch/cc-ci/main.go` **and** `/srv/cc-ci/cc-ci/main.go` — present in **both** clones, identical
> size, identical mtime … i.e. written by something outside git (**a git operation cannot leave the same
> untracked file in two clones**)."
**Repro:**
ls -la /srv/ # cc-ci -> /srv/cc-ci-orch (symlink)
stat -c '%d:%i %n' /srv/cc-ci-orch/cc-ci/main.go /srv/cc-ci/cc-ci/main.go
# -> 2049:3254604 for BOTH; stat -c 'links=%h' -> links=1
Same device, same inode, link count 1: **one file, one clone, seen through a symlink.** The same holds for the
Adversary clones (`/srv/cc-ci-orch/cc-ci-adv` and `/srv/cc-ci/cc-ci-adv` are inode `3206945`). The "two clones"
observation carries **zero** information about the writer, and the identical size/mtime are tautological.
**Risk bounded (read-only):** contents are a 281-byte hello-world `net/http` listener; `sha256
bdbc3bf167cd20f30c00880005f4f994f17f3722660973c9c65c7bf33e81ffaf`; no `go.mod`; **`go` is not installed**;
**nothing is listening on `:8080`**; `git log --all -- main.go` is empty; not gitignored; unreferenced by any
tracked file. It cannot execute. Present in the **Builder's** clone only — **absent from both Adversary
clone paths**, which is consistent with a single stray write into one working tree.
**Ask of the Builder:** drop the "both clones ⇒ outside git" inference (keep the file, keep the flag). Agreed
on not deleting it: neither of us created it.
---
### A-redfix-4 [adversary] — **CLOSED 2026-07-09T08:59Z** — all three asks applied at `e356698` and cold-re-verified by me at wake #45 (probes deleted; oscillation + silence recorded; precondition re-anchored to "never deploy `07fc6d4`"). The underlying *behaviour* remains open under **B-redfix-5**, which is deferred, not fixed. — the B-redfix-5 wedge has a **weekly autonomous re-trigger** (`nightly-sweep.timer`) and fails **silently**: post-arming, keycloak oscillates down-a-week / up-a-week with `nightly-sweep.service` reporting `Result=success`
Severity **MED** (operator-facing; describes a *deferred* item's blast profile, **not** a DoD item).
**No VETO. Does not reopen the phase.** M1 + M2 PASS stand. Merge target `b5f2b10` unchanged and still correct.
Supersedes the "absence of a re-trigger" half of **A-redfix-2** (my own error) and, with it, the text the
Builder accepted into STATUS-redfix.md at `f64d102`.
**Both prior descriptions of the wedge are wrong, in opposite directions.**
- Builder (`68b51d5`): "no code path redeploys it. **Recovery is manual.**" → wrong; it recovers unattended.
- Me (`b358b7e`, A-redfix-2): "no re-trigger … heals on the next **reboot / `nixos-rebuild switch`**" → wrong;
it re-triggers **weekly**, and an unrelated `switch` does *not* heal it.
**The re-trigger.** `warm_reconcile.py` has **two** invokers, not one. The second is the sweep:
nightly-sweep.timer OnCalendar=Sun *-*-* 03:00:00, Persistent=true
└─ nightly-sweep.service → /etc/cc-ci/runner/nightly_sweep.py (CCCI_REPO=/etc/cc-ci)
└─ main():147 roll_warm_infra() # unconditional after the _another_run_active() guard
└─ for app in WARM_APPS = ["keycloak", "traefik"]: # nightly_sweep.py:39,57
subprocess.run([python, warm_reconcile.py, app]) # nightly_sweep.py:59-61
**Observational proof that the sweep — not `warm-keycloak.service` — reconciles keycloak** (cc-ci, cold):
systemctl show warm-keycloak.service -p ExecMainStartTimestamp # → Wed 2026-06-17 17:29:31 UTC (once)
stat -c %y /var/lib/ci-warm/keycloak/last_good # → 2026-07-05 03:04:51.209 +0000
systemctl list-timers --all | grep nightly-sweep # → last trigger Sun 2026-07-05 03:04:50
`write_last_good()` is called **only** from `reconcile()`. The slot file was written **18 days after** the unit
last ran, and **1.2 s after** the sweep fired. ⇒ `reconcile("keycloak")` ran from the sweep. (`journalctl -u
nightly-sweep.service` retains 1 line — the empty grep is *retention*, not absence. Do not read it as evidence.)
**Why both of our probes missed it — the tell.** The invoking timer's name never contains the app name:
systemctl list-timers --all | grep -i keycloak # → EMPTY, yet a timer DOES drive it
git grep -n warm-keycloak -- nix/ | grep -iE 'timer|Restart=' # → EMPTY, yet a timer DOES drive it
Both are **unfalsifiable by construction**. STATUS-redfix.md (`f64d102`) currently prescribes *both* as the
verification commands for "no re-trigger". They must be removed. The sound probe is to grep for the **callee**:
git grep -n "warm_reconcile" -- runner/ nix/ | grep -v "^runner/warm_reconcile.py"
Same class as the `e3b0c44298fc1c14` empty-input tell (B-redfix-8) and the `restore()`/`docker` ordering tell:
a probe that returns the "all clear" value for a reason unrelated to the property under test.
**Corrected dynamics** (only once armed — i.e. only if a foreign `snapshot/meta.json` sits in the slot **and**
an upgrade is pending, `current != latest`):
| sweep | state on entry | branch taken | outcome |
|---|---|---|---|
| N | deployed, upgrade pending | stateful upgrade `:511-514` → `abra.undeploy` → `snapshot()` → guard raises | **keycloak DOWN** |
| N+1 | not deployed | fresh-deploy `:471-479` → redeploys `current`; **never calls `warmsnap`** | keycloak UP (old version) |
| N+2 | deployed, upgrade still pending | stateful upgrade again → guard raises | **keycloak DOWN** |
⇒ **keycloak alternates DOWN one week / UP the next**, indefinitely, until the foreign `snapshot/` is deleted
or `current == latest`. Not a one-shot outage, and not manual-recovery-required.
**And it is silent.** `roll_warm_infra()` captures the subprocess rc and **ignores it** (`nightly_sweep.py:59-63`
— `print(f"nightly: reconcile {app} rc={rc}")`, no raise), so a wedged reconcile does **not** fail
`nightly-sweep.service` (`Result=success`). The guard raises at `:514`, which is **outside** the try/except at
`:520-525` (that is B-redfix-5) and **upstream of every `write_alert()`** — so **no alert is written**. The only
trace is a journal line `nightly: reconcile keycloak rc=1`, in a unit whose output is not retained.
**A weekly production outage of the shared SSO provider would surface nowhere.**
**Good news — the trap cannot arm itself.** Arming requires the canonical seed to run from a tree that has the
keycloak enrollment but **not** the fix. The enrollment is **branch-only**:
git show origin/main:tests/keycloak/recipe_meta.py | grep WARM_CANONICAL # → False (= deployed /etc/cc-ci)
git show 07fc6d4:tests/keycloak/recipe_meta.py | grep WARM_CANONICAL # → True (enrollment, NO fix)
git show b5f2b10:tests/keycloak/recipe_meta.py | grep WARM_CANONICAL # → True (enrollment + fix)
The deployed checkout `/etc/cc-ci` carries `WARM_CANONICAL = False`, so **no autonomous node activity — including
Sunday's sweep — can write foreign meta into `/var/lib/ci-warm/keycloak/`.** Verified: the slot holds only
`last_good` (no `snapshot/`, no `canon-*`), and keycloak is currently deployed and healthy (`_app` + `_db`).
**⇒ The precondition is self-maintaining, and its landmark in STATUS is wrong.** It is not "the slot must be clean
*at merge time*" (a `git merge` executes nothing). The binding statement is:
> **Never deploy `07fc6d4`** — or any tree carrying the enrollment without `canonical_ns()`. That intermediate
> state is the *only* thing that can arm the trap. Merging `b5f2b10` ships enrollment **and** fix together, so
> canonical seeds land in `canon-keycloak/` and the slot never goes foreign.
**Secondary correction (`nixos-rebuild switch`).** `warm-keycloak.service`'s `ExecStart` embeds the runner by
store path — `/nix/store/…-runner/warm_reconcile.py`, and `warmsnap.py` lives in that **same** derivation
(`…-runner/harness/warmsnap.py`). So a switch restarts the unit **iff `runner/**` changed**. A switch that
touches anything else (e.g. sshd config) leaves the store path identical and does **not** re-run reconcile —
so "heals on `nixos-rebuild switch`" is false in general. Deploying the merge *does* change `runner/**`, which
means **the merge's own activation switch is itself a reconcile trigger**, alongside the Sunday sweep.
**Asks of the Builder (STATUS-redfix.md text only — no code change, nothing reopens):**
1. Delete both `grep -i keycloak` / `git grep warm-keycloak -- nix/` "no re-trigger" probes; they cannot fail.
2. Replace "recovery is manual" **and** "heals on reboot / `nixos-rebuild switch`" with the weekly
down/up oscillation above, and state that it is **silent** (`Result=success`, no `write_alert`).
3. Re-anchor the merge precondition from "slot clean at merge time" to "**never deploy `07fc6d4`**".
Remediation if ever armed is unchanged and still correct: **delete the foreign `snapshot/` from the slot**;
do not redeploy keycloak (the sweep does that for you, and it is what masks the fault).
---
### A-redfix-1 — ADDENDUM (wake #45, 2026-07-09T08:59Z): the embedded value is the **same live credential as B-redfix-8**, confirmed at value level; and rotation will **not** clean this file
Still **OPEN**. Re-verified cold on cc-ci this wake, without printing the secret:
ssh cc-ci 'stat -c "%a %U:%G" /etc/cc-ci/.git/config' # → 644 root:root
ssh cc-ci 'grep -oE "https://[^/@]+:[^/@]+@" /etc/cc-ci/.git/config \
| sed -E "s#https://[^:]+:##; s#@$##" | tr -d "\n" | sha256sum | cut -c1-16'
**EXPECTED / observed: `3fcea78925015fc9`** — byte-identical to the B-redfix-8 rotation sentinel
(`sha256(GITEA_PASSWORD)[:16]`, re-confirmed this wake from `/srv/cc-ci/.testenv` on the **orchestrator**).
So A-redfix-1 and B-redfix-8 are **two exposures of one still-unrotated, push-capable credential**, and the
sentinel matching proves it is **still not rotated**. (Extract-then-hash, never echo. `python3` is absent on
cc-ci — do not run the orchestrator's python probe over `ssh`; that is the `e3b0c44298fc1c14` empty-input tell.)
**New, and operator-relevant: rotation does not fix this file, and may recreate the exposure.**
`/etc/cc-ci` is a **manual `git clone`**, not nix-generated — `nix/hosts/cc-ci-hetzner/configuration.nix:7`
documents `git clone --recursive https://git.autonomic.zone/recipe-maintainers/cc-ci.git /etc/cc-ci`. Nothing
regenerates `.git/config` on `nixos-rebuild switch`. Therefore:
- After rotating `GITEA_PASSWORD`, the **old** value persists verbatim in this 0644 file (inert, but it should
still be scrubbed — it is the value published at `14c7dee`).
- If the operator re-clones or re-embeds the **new** password in the remote URL, the 0644 exposure **returns**.
**Recommendation (operator, alongside the B-redfix-8 rotation):** point the remote at a credential-less URL
(`https://git.autonomic.zone/recipe-maintainers/cc-ci.git`) and authenticate via a `credential.helper` /
token file with `0600`, or at minimum `chmod 0600 /etc/cc-ci/.git/config`. Do not carry userinfo in the URL.
**Rotation blast radius is small** (checked, so this is not a reason to delay): `GITEA_PASSWORD` is consumed
only by `scripts/bootstrap-drone-oauth.sh` (a one-off bootstrap). `scripts/recipe-mirror-sync.sh` pushes with
an **OAuth token**, not the password, so the weekly sweep's mirror sync does **not** depend on it.
Reference in New Issue View Git Blame Copy Permalink