autonomic-bot
3ec56d878e
Adversary confirmed all three claims and failed to refute "strip userinfo is safe" three ways: no push through /etc/cc-ci origin; secrets submodule anonymously fetchable so submodule update survives; nothing auto-pulls the checkout (/root/.git-credentials inert, no helper wired). Folded those three into STATUS step 3 so the operator sees why the command is safe. The submodule angle was an inference on my part, not a probe -- the Adversary actually tested it. Recorded in JOURNAL. Incidental (not a finding): cc-ci-secrets.git is public but SOPS-encrypted ciphertext. STATUS/JOURNAL text only. No code. ## DONE stands; M1+M2 PASS stand; no gate reopens.
632 lines
54 KiB
Markdown
632 lines
54 KiB
Markdown
# STATUS — phase `redfix`
|
||
|
||
Phase SSOT: `/srv/cc-ci/cc-ci-plan/plan-phase-redfix-canon-sweep-failures.md`
|
||
|
||
Mission: investigate every canon-sweep failure (discourse, mattermost-lts, mumble, bluesky-pds,
|
||
gitea, keycloak) → isolate → root-cause → classify (flake vs genuine; recipe vs test vs
|
||
warm-machinery vs load) → FIX each (recipe PR or harness improvement) → verify green. No standing
|
||
exceptions. Nothing merged.
|
||
|
||
## DONE — 2026-07-09T00:18Z
|
||
|
||
Phase `redfix` COMPLETE. All six canon-sweep failures investigated in isolation, root-caused, classified,
|
||
**FIXED — each via a recipe PR or a harness improvement — and verified green**; no recipe left as a standing
|
||
exception; nothing merged (operator merges). Every gate has a fresh Adversary PASS in REVIEW-redfix.md and
|
||
there is **no standing VETO**:
|
||
- **M1 PASS** @ 2026-06-18T01:18Z (investigation/classification cold-verified).
|
||
- **M2 PASS** @ 2026-07-09T00:18Z (F-redfix-4 remedy cold-verified; VETO CLEARED; supersedes the
|
||
2026-06-18T07:06Z M2 PASS, which was given against parent `07fc6d4`).
|
||
|
||
Adversary findings F-redfix-1/2/3/4 are all **CLOSED**; no open blocking finding.
|
||
|
||
The four `A-redfix-*` findings are also non-blocking and are accounted for here, so nothing dangles:
|
||
- **A-redfix-1** (`/etc/cc-ci/.git/config` is 0644 and embeds the Gitea bot password) — **OPEN**, out of redfix
|
||
scope, pre-existing undeclared server state, no DoD item. Mirrors my B-redfix-7. Not created by this phase.
|
||
**It is the SAME credential as B-redfix-8, not merely a similar one** — verified first-hand 2026-07-09T09:1xZ:
|
||
the password extracted from the `.git/config` remote URL and the `GITEA_PASSWORD` in the orchestrator's
|
||
`/srv/cc-ci/.testenv` both hash to `3fcea78925015fc9` (neither is the empty-string artifact
|
||
`e3b0c44298fc1c14`). **Two exposures, one credential, still unrotated.** Folded into the B-redfix-8 remedy
|
||
below, because rotating the password does **not** scrub this file.
|
||
- **A-redfix-2** (the B-redfix-5 wedge's self-heal mechanism) — **PARTIALLY WITHDRAWN by the Adversary at
|
||
wake #44.** Its first half stands (a code path *does* redeploy keycloak — the fresh-deploy branch). Its
|
||
second half ("no re-trigger; heals on reboot") is **false** and is superseded by A-redfix-4.
|
||
- **A-redfix-3** (the `main.go` "both clones" evidence) — **ACCEPTED**; that evidence is retracted below.
|
||
`/srv/cc-ci` is a symlink to `/srv/cc-ci-orch`. The file's origin remains unexplained; risk inert.
|
||
- **A-redfix-4** (a weekly `nightly-sweep.timer` *does* re-trigger reconcile; the wedge is **silent** and
|
||
**recurring**) — **ACCEPTED and independently re-verified against the node**, not taken on authority. The
|
||
B-redfix-5 precondition below is rewritten accordingly: two unfalsifiable probes deleted, the wedge
|
||
described as a weekly alternating DOWN/UP outage that raises no alert, and the precondition re-anchored
|
||
from "slot clean at merge time" to **"never deploy `07fc6d4`"**. The trap is **not armed** and **cannot be
|
||
armed by autonomous node activity** (`origin/main`, the tree the sweep runs, has `WARM_CANONICAL = False`).
|
||
|
||
All four corrected operator-facing *descriptions*, not DoD items. No VETO at wake #43 (`b358b7e`) or wake #44
|
||
(`84a1666`, 2026-07-09T08:47Z). **Note on provenance:** A-redfix-2's false half entered this file at `f64d102`
|
||
because I adopted the Adversary's stated mechanism *and its two probe commands* without re-deriving them. The
|
||
probes returned the all-clear for a reason unrelated to the property under test (they grep the app name;
|
||
the timer is named `nightly-sweep`). Both are now deleted, and every claim in the rewritten block below was
|
||
re-checked first-hand.
|
||
|
||
## ⚠ OPERATOR ACTION REQUIRED — B-redfix-8 (HIGH, open, outside DoD)
|
||
|
||
Phase DoD is met, but **one HIGH item is open and only an operator can close it.** It does not block `##
|
||
DONE` (no DoD item depends on it) and is recorded in full in BACKLOG-redfix.md.
|
||
|
||
**WHAT.** The live Gitea bot password (`GITEA_PASSWORD` in `/srv/cc-ci/.testenv`) was committed in
|
||
`14c7dee` (`machine-docs/BACKLOG-redfix.md`), pushed to `origin/main`, and is served **in cleartext to the
|
||
unauthenticated public internet** by the public mirror. It is push-capable to `recipe-maintainers/*`. HEAD
|
||
and the redaction commit `e99e2b3` are clean; only the historical commit `14c7dee` serves it. Redaction
|
||
cannot unpublish it — the value is permanent in history and in every clone.
|
||
|
||
**WHERE.** `git.autonomic.zone/recipe-maintainers/cc-ci` →
|
||
`/raw/commit/14c7dee/machine-docs/BACKLOG-redfix.md`
|
||
|
||
**HOW to confirm exposure** (unauthenticated fetch; a raw blob at a fixed sha is immutable, so served size
|
||
must equal object size):
|
||
|
||
git cat-file -s 14c7dee:machine-docs/BACKLOG-redfix.md # → 33408
|
||
|
||
An unauthenticated fetch of the raw URL returning **HTTP 200 / 33408 bytes** is the leak, not an error page.
|
||
|
||
**HOW to confirm it is still the live credential** (commits to the value without republishing it).
|
||
Run this **on the orchestrator** — `/srv/cc-ci/.testenv` lives there, **not** on `cc-ci`:
|
||
|
||
python3 -c 'import hashlib,re;v=re.search(r"^GITEA_PASSWORD=(.*)$",open("/srv/cc-ci/.testenv").read(),re.M).group(1).strip().strip(chr(34)).strip(chr(39));print(hashlib.sha256(v.encode()).hexdigest()[:16])'
|
||
|
||
**EXPECTED.** Prints `3fcea78925015fc9` while still unrotated. **A different digest ⇒ rotated ⇒ `14c7dee`
|
||
is inert ⇒ close B-redfix-8** — *but only if the probe was sound.* Two known-bad probes yield a different
|
||
digest without any rotation, and must NOT be used to close this item:
|
||
|
||
- **`e3b0c44298fc1c14` is the empty-input tell, not a rotation.** Running the command over `ssh cc-ci`
|
||
fails to open `.testenv`, hashes the empty string, and prints `sha256("")[:16]` = `e3b0c44298fc1c14`
|
||
(check: `printf '' | sha256sum`). Fix the probe; do not close B-redfix-8.
|
||
- **Do not conflate the three digests.** `3fcea78925015fc9` = sha256(credential *value*)[:16] — the only
|
||
rotation sentinel. `1994fd8d…` = sha256(the whole served *blob*) — immutable, changes never.
|
||
`e3b0c44298fc1c14` = sha256(empty) — a broken probe.
|
||
|
||
Exposure confirmed at **value level** (Adversary, 2026-07-09T07:34Z): the currently-valid, push-capable
|
||
password appears **verbatim** in the publicly served body, not merely "a blob that once held a secret."
|
||
|
||
**REMEDY (operator).** Rotate the Gitea bot password, then update **both** places that hold it. The credential
|
||
is a Class-A1 external infra input (plan §4.4) — the Builder must not rotate or invent it. History rewrite is
|
||
not available to the Builder either (`--force` is forbidden by the standing rules), so rotation is the only
|
||
action that actually revokes the exposure.
|
||
|
||
1. Rotate the bot password in Gitea.
|
||
2. Update `GITEA_PASSWORD` in `/srv/cc-ci/.testenv` **on the orchestrator**.
|
||
3. **Rewrite `/etc/cc-ci/.git/config` on `cc-ci` (A-redfix-1) — rotation does NOT scrub it.** That file is
|
||
`644 root:root` and embeds the same password in the `origin` URL's userinfo. `/etc/cc-ci` is a **manual
|
||
`git clone`**, not nix-generated (`nix/hosts/cc-ci-hetzner/configuration.nix:7` documents the clone), so no
|
||
`nixos-rebuild` regenerates or scrubs it. Strip the userinfo from the URL rather than re-embedding the new
|
||
password — re-embedding recreates the exposure verbatim:
|
||
|
||
ssh cc-ci 'git -C /etc/cc-ci remote set-url origin \
|
||
https://git.autonomic.zone/recipe-maintainers/cc-ci.git && chmod 600 /etc/cc-ci/.git/config'
|
||
|
||
Stripping the userinfo does **not** break the clone: the sweep only ever *fetches*, and the mirror serves
|
||
this repo anonymously (that is precisely what B-redfix-8 exploits). Verified 2026-07-09T09:1xZ —
|
||
`GIT_TERMINAL_PROMPT=0 git -c credential.helper= ls-remote https://git.autonomic.zone/recipe-maintainers/cc-ci.git HEAD`
|
||
→ `f0372b8… HEAD`, rc=0, no credential. The sibling `cc-ci-secrets` submodule URL (`.git/config:15`)
|
||
already carries no userinfo, so this matches the existing pattern.
|
||
|
||
**Independently cold-verified by the Adversary (REVIEW-redfix.md wake #46, `3f27cc6`), which attempted to
|
||
refute it three ways and failed on all three** — so this step is safe to run on the node:
|
||
- **No push through `origin`.** The only `git push` in the deployed tree (`scripts/recipe-mirror-sync.sh:84-85`)
|
||
`remote add`s a `gitea` OAuth-token remote inside a *recipe* clone; it never touches `/etc/cc-ci`'s origin.
|
||
- **The `secrets` submodule survives the strip.** Its URL already carries no userinfo and
|
||
`cc-ci-secrets.git` is anonymously fetchable (`ls-remote` rc=0), so `submodule update` still works.
|
||
- **Nothing auto-pulls the checkout.** `/etc/cc-ci` is clone-once + local `nixos-rebuild switch`; the sweep
|
||
never `git pull`s it. (`/root/.git-credentials` exists but no `credential.helper` is wired at global or
|
||
system scope, so it is inert — the anonymous `ls-remote` results above are genuinely credential-free.)
|
||
|
||
**Rotation blast radius is small — no operational reason to delay.** `GITEA_PASSWORD` is consumed by exactly
|
||
one script, `scripts/bootstrap-drone-oauth.sh` (`git grep -ln GITEA_PASSWORD -- .` returns that script, `docs/
|
||
install.md`, and machine-docs prose; nothing else in the runner). In particular `scripts/recipe-mirror-sync.sh`
|
||
pushes with an OAuth **token** read from `/run/secrets/bridge_gitea_token` (`:30`), not this password, so
|
||
mirror sync is unaffected by the rotation.
|
||
|
||
**HOW to confirm BOTH exposures are closed** (each prints a digest; a value other than `3fcea78925015fc9` on
|
||
both, and not the empty-string tell `e3b0c44298fc1c14`, means rotated):
|
||
|
||
# exposure 2 — the embedded copy; `python3` is absent on cc-ci, so use sha256sum
|
||
ssh cc-ci 'sed -n "s#^\s*url = https://[^:]*:\([^@]*\)@git\.autonomic\.zone/recipe-maintainers/cc-ci\.git#\1#p" \
|
||
/etc/cc-ci/.git/config | head -1 | tr -d "\n" | sha256sum | cut -c1-16'
|
||
|
||
**EXPECTED after step 3:** the command prints **nothing** (no userinfo left to match) — an empty result here
|
||
is the fix, whereas an `e3b0c44298fc1c14` from the *step-2* probe is a broken probe. Do not confuse them.
|
||
|
||
**Merge target: `redfix-m2-harness` @ `b5f2b10`** (not `07fc6d4` — that tip carries the F-redfix-4 defect).
|
||
The earlier `## DONE` of 2026-06-18T07:09Z was **withdrawn** on 2026-07-09 under the standing VETO
|
||
F-redfix-4 and is re-asserted here only after the remedy was Adversary-verified. Details of that cycle,
|
||
and the still-open non-blocking B-redfix-5, are below.
|
||
|
||
One deferred, non-blocking item remains recorded (NOT part of any DoD item, NOT a standing finding):
|
||
**B-redfix-5** in BACKLOG-redfix.md — in `warm_reconcile.py`, TWO post-`abra.undeploy()` calls sit outside
|
||
the upgrade's `try/except`: `warmsnap.snapshot()` on the upgrade path (`:514`) and `warmsnap.restore()` on
|
||
the rollback path (`:536`). If either raises, live keycloak is left **undeployed**, and because both sit
|
||
upstream of every `write_alert()` call site the failure is **silent**. Pre-dates the enrollment (present at
|
||
`07fc6d4`). Adversary concurred it is not a VETO. Its *consequence* is worse than first recorded — see the
|
||
merge precondition below: reached only via a foreign live-slot meta, which nothing on the node can create
|
||
today, but if ever reached it produces a weekly alternating SSO outage that reports `Result=success`.
|
||
|
||
**MERGE PRECONDITION for `b5f2b10`.** The F-redfix-4 fix *adds* a raise path to both of those calls —
|
||
`warmsnap._assert_slot_not_foreign()` (`warmsnap.py:158` in `snapshot()`, `:209` in `restore()`). It raises
|
||
iff the live slot's `snapshot/meta.json` records a `domain` other than the one passed. Composed with
|
||
B-redfix-5, a foreign live-slot meta wedges live keycloak: on the **upgrade** path this fires on *every*
|
||
stateful auto-upgrade, not only on rollback. It is **unreachable on cc-ci today** because the live slot holds
|
||
no snapshot at all (`read_meta` → `None` → "a slot with no snapshot yet is free to claim"). It is unreachable
|
||
for that reason — **not** because F-redfix-4 is closed.
|
||
|
||
**THE BINDING RULE: never deploy `07fc6d4`.** (Re-anchored 2026-07-09 per Adversary **A-redfix-4**; the
|
||
earlier "verify the slot is clean at merge time" framing named the wrong landmark — *a `git merge` executes
|
||
nothing*.) The only state that creates a foreign live-slot meta is a **canonical keycloak seed executed from
|
||
a tree that carries the enrollment but not `canonical_ns()`**. Exactly one such tree exists: `07fc6d4`
|
||
(`tests/keycloak/recipe_meta.py` → `WARM_CANONICAL = True`, no `canonical_ns()`). `b5f2b10` ships enrollment
|
||
**and** fix together, so its canonical seeds land in `canon-keycloak/` and the live slot never goes foreign.
|
||
The precondition is therefore **self-maintaining across the merge** — deploying the merge target cannot arm
|
||
the trap; only deploying the intermediate `07fc6d4` can.
|
||
|
||
**No autonomous node activity can arm it.** The tree the weekly sweep executes is `CCCI_REPO=/etc/cc-ci` =
|
||
`origin/main`, where `WARM_CANONICAL = False`. Only `07fc6d4` and `b5f2b10` have `True`.
|
||
|
||
**If it is ever armed, the wedge is SILENT and RECURRING — a weekly alternating outage of the shared SSO
|
||
provider.** (Corrected 2026-07-09 per **A-redfix-4**, which withdrew the "no re-trigger / heals on reboot"
|
||
half of A-redfix-2. My own earlier "recovery is manual" (`68b51d5`) was also wrong. Re-derived independently
|
||
against the node — see HOW below.) `warm_reconcile.py` has **two** invokers, and the second is a weekly timer:
|
||
|
||
nightly-sweep.timer (OnCalendar=Sun *-*-* 03:00:00, Persistent=true)
|
||
└─ nightly_sweep.main():147 → roll_warm_infra() → subprocess warm_reconcile.py keycloak
|
||
(WARM_APPS = ["keycloak", "traefik"], nightly_sweep.py:39,57-61)
|
||
|
||
Once armed, with `current != latest`: **sweep N** → `abra.undeploy()` → `snapshot()` guard raises → keycloak
|
||
**DOWN**. **Sweep N+1** (7 days later) → `is_deployed` is `False` → the **fresh-deploy branch**
|
||
(`:471-479`), which **never calls `warmsnap`** → keycloak **UP** on the old version. **Sweep N+2** → upgrade
|
||
path → **DOWN** again. **Alternating weeks, indefinitely**, because the foreign `snapshot/meta.json` is never
|
||
removed. (`abra.undeploy()` (`abra.py:295-297`) runs only `abra app undeploy -n` and does **not** remove the
|
||
app `.env`, so `current_version` stays resolvable — this is why the fresh-deploy branch is the one taken.)
|
||
|
||
**And it raises no alert.** `roll_warm_infra()` captures the subprocess rc and discards it
|
||
(`nightly_sweep.py:59-62` — `rc` is only ever printed), so `nightly-sweep.service` still reports
|
||
`Result=success`. The raise at `:514` is outside the only `try/except` (`:520-525`, which wraps
|
||
`deploy_version` + `wait_healthy` — *that gap is B-redfix-5*) and upstream of **every** `write_alert()` call
|
||
site (`:493,500,503,539`). `reconcile()`'s sole caller `main():556` does not catch either. A weekly SSO
|
||
outage would surface **nowhere**. (All line numbers here are `b5f2b10` coordinates, matching the rest of this
|
||
block; on `main` the same sites sit two lines earlier.)
|
||
|
||
**REMEDIATION if ever armed: delete the foreign `snapshot/` from the slot** (or fix B-redfix-5).
|
||
**Redeploying keycloak does NOT fix it** — the next sweep re-wedges. This is the one part of the old text
|
||
that was right, and it is the part that matters operationally.
|
||
|
||
**HOW to verify** (from a clone at `b5f2b10`). Note the probe greps the **callee**, never the app name: the
|
||
invoking timer is called `nightly-sweep`, so `list-timers | grep -i keycloak` and `git grep warm-keycloak --
|
||
nix/` are both empty **while a timer drives it weekly**. Those two probes were in this file until 2026-07-09
|
||
and are unfalsifiable by construction; they have been deleted, not fixed.
|
||
|
||
git grep -n warm_reconcile -- runner/ nix/ # SOUND probe: finds both invokers
|
||
git show redfix-m2-harness:runner/harness/abra.py | sed -n '295,297p' # undeploy: no .env removal
|
||
git show redfix-m2-harness:runner/warm_reconcile.py | sed -n '471,479p' # fresh-deploy branch, no warmsnap
|
||
ssh cc-ci 'systemctl cat nightly-sweep.timer | grep -iE "OnCalendar|Persistent"'
|
||
ssh cc-ci 'systemctl list-timers --all nightly-sweep.timer --no-pager'
|
||
|
||
**EXPECTED:** `git grep warm_reconcile` prints both `nix/modules/warm-keycloak.nix:26` (the oneshot unit) and
|
||
`runner/nightly_sweep.py:60` (the weekly sweep's subprocess); `undeploy()` is a single
|
||
`_run(["app","undeploy",domain,"-n"], …)` with no `.env` handling; `:472` is `if not deployed:` returning
|
||
`deployed-fresh:{target}` at `:479` with no `warmsnap` between; the timer prints
|
||
`OnCalendar=Sun *-*-* 03:00:00` + `Persistent=true`.
|
||
|
||
**Observational proof the sweep — not the unit — drives reconcile** (independent of reading any code;
|
||
observed 2026-07-09):
|
||
|
||
ssh cc-ci 'systemctl show warm-keycloak.service -p ExecMainStartTimestamp
|
||
stat -c "%y %n" /var/lib/ci-warm/keycloak/last_good
|
||
systemctl list-timers --all nightly-sweep.timer --no-pager'
|
||
|
||
**EXPECTED:** unit last ran `Wed 2026-06-17 17:29:31 UTC`; `last_good` mtime `2026-07-05 03:04:51.209`; timer
|
||
last trigger `Sun 2026-07-05 03:04:50 UTC`. The slot was written **18 days after the unit last ran and 1.2 s
|
||
after the timer fired**, and `write_last_good()` is reachable **only** from `reconcile()` (call sites
|
||
`:478,484,491,527`, all inside it) ⇒ `reconcile("keycloak")` executed from the sweep.
|
||
|
||
**Also:** `warm-keycloak.service`'s `ExecStart` resolves to `/nix/store/…-runner/warm_reconcile.py`, and
|
||
`harness/warmsnap.py` lives in that **same** derivation. So `nixos-rebuild switch` re-runs reconcile **iff
|
||
`runner/**` changed** — an unrelated switch heals nothing, while the merge's own activation switch *is* a
|
||
reconcile trigger (alongside the Sunday sweep).
|
||
|
||
**Blast radius: keycloak only.** `warm.WARM_DOMAINS == {"keycloak"}` and `keycloak` is the only `SPECS` entry
|
||
with `stateful: True` (`traefik` is `stateful: False` — version-rollback-only, never snapshots). No other
|
||
warm unit reaches either call site.
|
||
|
||
**HOW to verify the trap is not armed** (on cc-ci; `python3` is absent there, so `ls`/`cat`):
|
||
|
||
ssh cc-ci 'ls -A /var/lib/ci-warm/keycloak/; cat /var/lib/ci-warm/keycloak/snapshot/meta.json 2>&1'
|
||
|
||
**EXPECTED (observed 2026-07-09T08:5xZ):** `last_good` only, and `cat` reports `No such file or directory` —
|
||
no `snapshot/`, hence no `meta.json`; no `/var/lib/ci-warm/canon-keycloak` either; live keycloak `200`. **A
|
||
`meta.json` whose `"domain"` is anything other than `warm-keycloak.ci.commoninternet.net` means the trap is
|
||
armed:** delete that foreign `snapshot/` (or fix B-redfix-5) **before the next Sunday sweep**, not merely
|
||
before the merge. A `meta.json` recording `warm-keycloak.ci.commoninternet.net` is self-consistent and passes
|
||
the guard.
|
||
|
||
---
|
||
|
||
### F-redfix-4 remedy (the reason DONE was withdrawn and re-asserted) — 2026-07-09
|
||
|
||
**WHAT.** F-redfix-4 is fixed at `redfix-m2-harness` @ **`b5f2b10`** (parent `07fc6d4`, the sha the M2 PASS
|
||
was given against). Warm state is now keyed by a **stack namespace**, not a bare recipe:
|
||
`canonical.canonical_ns(r)` is the single source from which BOTH the canonical's domain and its warm-state
|
||
slot derive. A live-warm provider (`r in warm.WARM_DOMAINS`) gets ns `canon-<r>` → domain
|
||
`warm-canon-keycloak.ci.commoninternet.net` (**unchanged**) + slot `/var/lib/ci-warm/canon-keycloak/`.
|
||
Every other recipe keeps ns `<r>` (the invariant is structural — `r not in warm.WARM_DOMAINS` — not a
|
||
property of any particular count). `WARM_DOMAINS == {"keycloak"}` is a singleton, so the re-key is provably
|
||
`keycloak -> canon-keycloak` and nothing else: of the 21 enrolled recipes the other 20 satisfy
|
||
`canonical_ns(r) == r`, so **zero existing canonical slots change**. On live disk that is 17 seeded
|
||
canonicals (those carrying a `canonical.json`), all of which remain in `prune_stale`'s `keep` set.
|
||
|
||
Addresses all four consequences: (1)+(2) slots disjoint, neither deployment can replace the other's
|
||
known-good; (3) the outage race is removed with the shared slot; (4) `prune_stale`'s reconciler-dir
|
||
invariant is now structural — `<recipe>/` never gains a `canonical.json`.
|
||
|
||
Also: `warmsnap._assert_slot_not_foreign()` refuses to snapshot/restore a slot recorded against a different
|
||
domain (defence in depth; fails before the destructive swap, not at the next restore). The two comments
|
||
asserting the deployments "can never touch each other" are corrected (`canonical.py`,
|
||
`tests/keycloak/recipe_meta.py`). `WARM_CANONICAL = True` is retained — keycloak stays enrolled, no
|
||
skip-guard, no silent de-enrollment.
|
||
|
||
**MIGRATION: none required.** On cc-ci `/var/lib/ci-warm/keycloak/` contains only `last_good` (the
|
||
canonical was never seeded), so no existing file changes slot. Verify: `ls -A /var/lib/ci-warm/keycloak/`
|
||
→ `last_good` only.
|
||
|
||
**WHERE.** cc-ci repo, branch `redfix-m2-harness` @ `b5f2b10`. Files: `runner/harness/warmsnap.py`,
|
||
`runner/harness/canonical.py`, `runner/warm_reconcile.py`, `runner/run_recipe_ci.py`,
|
||
`tests/keycloak/recipe_meta.py`, `tests/unit/test_warmsnap.py`, `tests/unit/test_canonical.py`.
|
||
|
||
**HOW to verify (1) — unit suite, cold clone, no docker needed.**
|
||
|
||
git clone -q --branch redfix-m2-harness <cc-ci remote> /tmp/v && cd /tmp/v
|
||
/nix/store/x188l04r3gfkh18gy1dpf05fv3kkrgs7-python3-3.12.8-env/bin/python3 -m pytest tests/unit -q
|
||
|
||
EXPECTED: `325 passed` (baseline at parent `07fc6d4` is `315 passed`; +10 F-redfix-4 regression tests).
|
||
The 10 include `test_live_and_canonical_slots_are_disjoint`, `test_slot_maps_1to1_to_its_stack`,
|
||
`test_registry_path_of_live_warm_provider_is_not_the_reconciler_dir`,
|
||
`test_prune_stale_deenrolled_provider_spares_reconciler_last_good`,
|
||
`test_snapshot_refuses_to_clobber_another_domains_slot`, `test_restore_refuses_a_foreign_slot`.
|
||
|
||
**HOW to verify (2) — the VETO's clearing condition, on the real node.** Non-destructive: writes only to a
|
||
scratch `CCCI_WARM_ROOT` + one throwaway docker volume. Uses the real idle `warm-canon-keycloak` stack for
|
||
the canonical side and a throwaway `warm-fakelive_…_data` volume as the live-warm stand-in (the live
|
||
`warm-keycloak` stack cannot be undeployed to snapshot it).
|
||
|
||
ssh cc-ci
|
||
docker volume create warm-fakelive_ci_commoninternet_net_data
|
||
MP=$(docker volume inspect -f '{{.Mountpoint}}' warm-fakelive_ci_commoninternet_net_data); echo x > $MP/live.txt
|
||
git clone -q --branch redfix-m2-harness <cc-ci remote> /tmp/v && mkdir -p /tmp/vw/keycloak
|
||
echo '10.7.1+26.6.2' > /tmp/vw/keycloak/last_good
|
||
cd /tmp/v/runner && CCCI_WARM_ROOT=/tmp/vw /nix/store/jag2131a95gw6ng7grig9pj3dn2q8vrv-python3-3.12.8-env/bin/python3 - <<'PY'
|
||
import sys; sys.path.insert(0, ".")
|
||
from harness import warmsnap as ws, canonical as c
|
||
LIVE, CANON = ws.live_slot("keycloak"), c.canonical_slot("keycloak")
|
||
CANON_DOM, LIVE_DOM = c.canonical_domain("keycloak"), "warm-fakelive.ci.commoninternet.net"
|
||
print(ws.snap_dir(LIVE), ws.snap_dir(CANON), c.registry_path("keycloak"), sep="\n")
|
||
ws.snapshot(CANON, CANON_DOM, version="canon-known-good")
|
||
ws.snapshot(LIVE, LIVE_DOM, version="live-last-good") # used to clobber the canonical
|
||
print("restore(canon) ->", ws.restore(CANON, CANON_DOM)["volumes"])
|
||
print("restore(live) ->", ws.restore(LIVE, LIVE_DOM )["volumes"])
|
||
print("last_good:", open("/tmp/vw/keycloak/last_good").read().strip())
|
||
try: ws.snapshot(CANON, LIVE_DOM); print("FAIL: foreign snapshot allowed")
|
||
except ws.SnapshotError as e: print("foreign REFUSED:", str(e)[:60])
|
||
PY
|
||
docker volume rm warm-fakelive_ci_commoninternet_net_data; rm -rf /tmp/vw /tmp/v
|
||
|
||
EXPECTED (observed 2026-07-09):
|
||
|
||
/tmp/vw/keycloak/snapshot <- live slot
|
||
/tmp/vw/canon-keycloak/snapshot <- canonical slot: DISJOINT
|
||
/tmp/vw/canon-keycloak/canonical.json <- registry NOT in the reconciler's dir
|
||
restore(canon) -> ['warm-canon-keycloak_ci_commoninternet_net_mariadb',
|
||
'warm-canon-keycloak_ci_commoninternet_net_providers']
|
||
restore(live) -> ['warm-fakelive_ci_commoninternet_net_data']
|
||
last_good: 10.7.1+26.6.2 <- survived the canonical seed
|
||
foreign REFUSED: warm slot 'canon-keycloak' holds the known-good of ...
|
||
|
||
Each `restore()` returns its OWN stack's volumes (the clearing condition). Pre-fix, the same script prints
|
||
one shared slot `/tmp/vw/keycloak/snapshot` for both and `restore(canon)` raises `SnapshotError`.
|
||
|
||
**Node integrity after my own run of the above (2026-07-09).** Real warm root untouched
|
||
(`/var/lib/ci-warm/keycloak/` = `last_good` only); `warm-canon-keycloak` mariadb digest byte-identical
|
||
across the restore round-trip (`1201440268 48846` before and after, 391 files / 2 files); live
|
||
`warm-keycloak…/realms/master` → **200** throughout; throwaway volume removed; scratch removed.
|
||
|
||
**Scope.** keycloak only. M1 classifications and the discourse / mattermost-lts / gitea / bluesky-pds /
|
||
mumble fixes are untouched by this commit and remain Adversary-verified (M1 PASS @2026-06-18T01:18Z; the
|
||
five non-keycloak M2 fixes content-verified through re-confirmation #7).
|
||
|
||
**Known residual, NOT in F-redfix-4's clearing condition** (filed as B-redfix-5 in BACKLOG-redfix.md, not
|
||
blocking): `warm_reconcile.py`'s rollback `warmsnap.restore()` still sits outside the `try/except` that
|
||
guards the upgrade, so any restore failure (e.g. a corrupt/absent snapshot) leaves live keycloak undeployed
|
||
after `abra.undeploy()`. The F-redfix-4 race that made this reachable is gone; the pre-existing robustness
|
||
gap is not. Recorded, not silently dropped.
|
||
|
||
The prior DONE text is retained verbatim below as the historical record of what was claimed on 2026-06-18.
|
||
|
||
---
|
||
|
||
## (HISTORICAL — withdrawn 2026-07-09 under VETO F-redfix-4; superseded by the DONE above) DONE — 2026-06-18T07:09Z
|
||
|
||
Phase `redfix` COMPLETE. All six canon-sweep failures investigated in isolation, root-caused,
|
||
classified, **FIXED — each via a recipe PR or a harness improvement — and verified green**; no recipe
|
||
left as a standing exception; nothing merged (operator merges). Both gates have a fresh Adversary PASS
|
||
in REVIEW-redfix.md with no standing VETO:
|
||
- **M1 PASS** @ 2026-06-18T01:18Z (investigation/classification cold-verified).
|
||
- **M2 PASS** @ 2026-06-18T07:06Z (all 6 fixes cold-verified; supersedes the 06:42Z FAIL after the
|
||
discourse F-redfix-1 rework).
|
||
|
||
Fixes (per recipe): mattermost-lts recipe PR #1 (pg_backup.sh + restore.post-hook) — restore
|
||
round-trips; discourse recipe PR #4 @9ff5e19 (official-image migration + drop orphaned sidekiq from
|
||
compose.smtpauth.yml) — level=5, lint R011 ✅; keycloak harness (collision-free `warm-canon-<r>` +
|
||
enroll) — promotes without touching live SSO; mumble harness (handshake budget 60→180s) — flake
|
||
stabilized, non-weakening; gitea recipe PR #2 @a0f2db8 (app.ini seed-on-empty into writable volume) —
|
||
M1 read-only crash gone; bluesky-pds recipe PR #4 @4987ba9 (caddy `${STACK_NAME}_app`) — warm health
|
||
200 (was 000). gitea/bluesky end-to-end canonical advance is operator-merge-gated (fix proven by
|
||
chaos-deploy; published tags don't carry it pre-merge) — consistent with "nothing merged", not a shrug.
|
||
|
||
### Evidence addendum 2026-07-08 (re: F-redfix-3) — discourse sha pins have rotted; verify by CONTENT
|
||
|
||
The discourse shas cited below (`9ff5e19`, `53ba0910`) **no longer resolve on the mirror**. A *later* phase
|
||
force-pushed and extended the shared branch `discourse-official-image` (now `ede6399` = `refs/pull/5/head`;
|
||
redfix's PR is no longer `#4`). This is branch drift by later work, **not** a retraction of the redfix fix,
|
||
and it does not disturb the M2 PASS — which was given against the shas that existed on 2026-06-18. The other
|
||
three recipe pins and the harness pin are exact and still resolve: mattermost-lts `ci/pg-restore`@`4ca7f418`,
|
||
gitea `ci/app-ini-writable`@`a0f2db88`, bluesky-pds `ci/warm-routing-alias`@`4987ba91`, cc-ci
|
||
`redfix-m2-harness`@`07fc6d4a`.
|
||
|
||
Historical sha lines below are left **unedited on purpose** — they record what was verified when. Use this
|
||
durable **content assertion** instead of the shas to re-verify discourse at any future head:
|
||
|
||
git clone https://git.autonomic.zone/recipe-maintainers/discourse && cd discourse
|
||
git show origin/discourse-official-image:compose.yml | grep -m1 'image:.*discourse'
|
||
git show origin/discourse-official-image:compose.smtpauth.yml | grep -c sidekiq
|
||
|
||
EXPECTED: `image: discourse/discourse:3.5.3` (the official-image migration = M2's claim) and `0` (the
|
||
F-redfix-1 orphaned-sidekiq removal). Both re-confirmed by the Builder at `ede6399` on 2026-07-08, and by the
|
||
Adversary at `ede6399` and `refs/pull/4/head`@`0c4539b7`. (Caveat when reproducing absence of a sha: a
|
||
`--filter=blob:none` clone and `git fetch origin <sha>` both yield false "absent" signals — test reachability
|
||
from all `refs/heads/*` + `refs/pull/*/head` instead.)
|
||
|
||
---
|
||
|
||
## Phase: M1 — investigate + isolate + classify (IN PROGRESS)
|
||
|
||
Bootstrapped 2026-06-17T23:20Z. cc-ci healthy, no run in flight, next scheduled sweep 2026-06-21
|
||
(3-day clear window). Disk `/` 38G free (75% used).
|
||
|
||
### Isolation harness (how I reproduce each failure ALONE)
|
||
|
||
Each canon-sweep per-recipe run is `runner/nightly_sweep.run_on_tag(recipe, latest)`:
|
||
`abra.recipe_checkout(recipe, <latest-tag>)` then `run_recipe_ci.py` with `RECIPE=<r>
|
||
CCCI_SKIP_FETCH=1` and REF/QUICK/MODE/VERSION unset (cold, full, head==tag). Isolation = run ONE
|
||
recipe at a time with NO concurrent sweep load on the single node (the loaded node is the known
|
||
flake source per phase plan §2.1). Runs execute on cc-ci from `/etc/cc-ci`.
|
||
|
||
### Starting canonical state (cc-ci `/var/lib/ci-warm/<r>/canonical.json`, read 2026-06-17T23:19Z)
|
||
|
||
| Recipe | Canonical now | Note |
|
||
|---|---|---|
|
||
| discourse | (none) | no canonical dir |
|
||
| mattermost-lts | (none) | no canonical dir |
|
||
| mumble | `1.0.0+v1.6.870-0` @ 20260617T180501Z | **canonical PRESENT, written TODAY** — flake signal |
|
||
| bluesky-pds | (none) | no canonical dir |
|
||
| gitea | `3.5.3+1.24.2-rootless` @ 20260617T083930Z | 3.6.0 advance not promoted |
|
||
| keycloak | (none) | de-enrolled (WARM_CANONICAL off) |
|
||
|
||
### M1 investigation tracker
|
||
|
||
| Recipe | Isolation run | Result | Root cause | Classification |
|
||
|---|---|---|---|---|
|
||
| discourse | DONE @23:40Z (`/tmp/redfix-discourse.log` on cc-ci) | install/backup/restore/custom PASS; **upgrade overlay FAIL**. Deploys+serves fine — NOT a timeout/FATA. | cc-ci overlay `tests/discourse/test_upgrade.py` asserts head runs official `discourse/discourse:3.5.3` + drops sidekiq; latest tag `0.8.1+3.5.0` AND main both still `bitnamilegacy/discourse:3.5.0`+sidekiq (migration exists in no release/main). The `depends_on discourse` string is a non-fatal prepull-only warning, not the deploy. | **stale/PR-specific cc-ci OVERLAY test** mismatched to canonical-sweep context (not flake/timeout/recipe-deploy/warm-machinery) |
|
||
| mattermost-lts | DONE @00:05Z (`/tmp/redfix-mattermost-lts.log`) | install/upgrade/backup/custom PASS; **restore FAIL** `ci_marker does not exist` — **deterministic in isolation** (not a load race) | recipe `postgres` svc backup labels: backs up hot live PGDATA + dump but has **NO `backupbot.restore.post-hook`** to replay the dump → restore doesn't round-trip postgres. Contrast immich (passes): dump-only `backup.volumes.postgres.path: backup.sql` + `restore.post-hook: /pg_backup.sh restore`. | **genuine RECIPE defect** at latest → recipe PR (adopt immich-style dump+restore-post-hook) |
|
||
| mumble | DONE — **2× isolation GREEN** (`/tmp/redfix-mumble.log` + `/tmp/redfix-mumble2.log`) | **ALL tiers PASS** incl. handshake on BOTH runs; no orphans; canonical re-promoted green each time | handshake (TLS+ServerSync) not completing within ~60s retry under heavy concurrent sweep load; fine in isolation | **load/timing FLAKE** → harness stabilization (readiness gate / retry) |
|
||
| bluesky-pds | DONE @00:45Z (`/tmp/redfix-bluesky-pds.log` + live diag) | cold lifecycle GREEN; **WC5 promote 000** reproduces (warm /xrpc/_health last status 0). NOT a flake | caddy on-demand TLS (`ask http://app:3000/tls-check`) can't reach app: caddy resolves bare `app` to OTHER stacks' app endpoints on shared `proxy` net (getent app→only 10.10.0.X, never internal 10.0.3.3; proxy has drone/traefik/keycloak/ccci `app` aliases) → no cert → 000. Promote machinery correct (refused to write canonical). | **genuine routing/RECIPE defect** (cross-stack `app`-alias collision on shared proxy) → recipe PR: unique PDS service name/alias. NOT promote-machinery, NOT flake |
|
||
| gitea | DONE @00:14Z (`/tmp/redfix-gitea2.log` + live container logs) | cold lifecycle (incl fresh 3.5.3→3.6.0 upgrade) PASS; **warm advance crash-loops** | `LoadCommonSettings() [F] error saving JWT Secret … failed to save "/etc/gitea/app.ini": read-only file system` — gitea 3.6.0/1.24.2 tries to persist a JWT to the read-only app.ini docker-config mount on warm reattach (before DB migration; 3.5.3 data intact). Cold passes (fresh secrets, no rewrite). | **genuine RECIPE defect** (3.6.0 + read-only app.ini config mount on advance) → recipe PR: render app.ini into the writable config volume. (1st gitea run hit a nixenv "already deployed" leftover confound — fixed by undeploying to idle then re-running) |
|
||
| keycloak | DONE @01:05Z (code-verified; no run) | de-enrolled. `canonical_domain("keycloak")` == `WARM_DOMAINS["keycloak"]` == `warm-keycloak.ci.commoninternet.net` EXACTLY (canonical.py:42, warm.py:27,44). Live keycloak 200 /realms/master. | data-warm canonical domain uses same `warm-<r>` scheme as the live-warm OIDC provider → promote would collide with live shared SSO. No collision-free canonical namespace exists. | **HARNESS defect** (warm-domain namespace collision) → fix: collision-free `canonical_domain` for live-warm providers (`warm-canon-<r>`), then enroll keycloak |
|
||
|
||
### M1 results table (recipe → failure → isolation result → root cause → classification → fix approach)
|
||
|
||
| Recipe | Canon-sweep failure | Isolation result | Flake or genuine | Root cause | Class | Fix approach (M2) |
|
||
|---|---|---|---|---|---|---|
|
||
| discourse | "cold-deploy timeout / deploy FATA" | install/backup/restore/custom GREEN; **upgrade overlay RED** | **genuine (deterministic)** — but the canon root-cause was WRONG (no timeout, no deploy FATA) | cc-ci overlay `tests/discourse/test_upgrade.py` asserts head = official `discourse/discourse:3.5.3` + sidekiq dropped; that migration is in NO release tag and NOT in main (all use `bitnamilegacy/discourse:3.5.0`+sidekiq) | **stale/PR-specific cc-ci OVERLAY test** | make the overlay assert migration-faithfulness only when the head IS that migration (not vs a release tag), OR a recipe PR migrating off deprecated bitnamilegacy — settle in M2 (NOT a test-weakening) |
|
||
| mattermost-lts | `test_restore_returns_state` RED | install/upgrade/backup/custom GREEN; **restore RED** (`ci_marker does not exist`) | **genuine (deterministic in isolation)** — NOT the canon "loaded-node race" | recipe postgres backup labels back up hot PGDATA + a dump but have **no `backupbot.restore.post-hook`** to replay it; restore doesn't round-trip. immich (passes) uses dump-only path + `restore.post-hook` | **genuine RECIPE defect** | recipe PR: adopt immich-style postgres dump + `backupbot.restore.post-hook` replay |
|
||
| mumble | `test_handshake…` RED | **ALL tiers GREEN** in isolation (×N) incl. handshake | **FLAKE (load/timing)** | handshake (TLS+ServerSync) doesn't complete within the 60s retry under heavy concurrent sweep load; fine isolated; canonical written green today | **load/concurrency FLAKE** | harness stabilization: stronger readiness gate before the custom tier / longer-or-smarter handshake retry |
|
||
| bluesky-pds | warm promote `/xrpc/_health` → 000 | cold lifecycle GREEN; **warm promote 000 reproduces** | **genuine (deterministic)** — NOT a load/rate-limit flake | caddy on-demand TLS calls `http://app:3000/tls-check`; caddy resolves bare `app` to OTHER stacks' app endpoints on the shared `proxy` net (every stack aliases its main svc `app`), never bluesky's own internal app (10.0.3.3) → connection refused → no cert → 000 | **genuine ROUTING/RECIPE defect** (cross-stack `app`-alias collision) | recipe PR: give the PDS service a unique name/alias so caddy resolves only bluesky's app |
|
||
| gitea | 3.5.3→3.6.0 warm advance doesn't promote | cold (incl fresh upgrade) GREEN; **warm advance crash-loops** | **genuine (deterministic)** | gitea 3.6.0/1.24.2 saves a JWT secret to `/etc/gitea/app.ini` on warm reattach; app.ini is a **read-only docker-config mount** → `read-only file system` FATA at LoadCommonSettings (pre-migration; 3.5.3 data intact). Cold passes (fresh secrets, no rewrite) | **genuine RECIPE defect** | recipe PR: render app.ini into the writable `config:/etc/gitea` volume (entrypoint) instead of a read-only docker config |
|
||
| keycloak | de-enrolled (not tested) | code-verified (no run) | **genuine (structural)** | `canonical_domain("keycloak")` == `WARM_DOMAINS["keycloak"]` == `warm-keycloak.ci.commoninternet.net` EXACTLY → a data-warm canonical would collide with the live-warm OIDC provider | **HARNESS defect** (warm-domain namespace collision) | harness: collision-free `canonical_domain` for live-warm providers (`warm-canon-<r>`), then enroll keycloak (WARM_CANONICAL=True) |
|
||
|
||
### HOW the Adversary cold-verifies each classification (run ONE recipe at a time, no concurrent load)
|
||
|
||
Isolation invocation (per recipe `R` at latest tag `T`), from `/etc/cc-ci` on cc-ci:
|
||
`git -C ~/.abra/recipes/R checkout -f --quiet T && env -u REF -u CCCI_QUICK -u MODE -u VERSION RECIPE=R CCCI_SKIP_FETCH=1 cc-ci-run runner/run_recipe_ci.py`
|
||
Latest tags: discourse `0.8.1+3.5.0`, mattermost-lts `2.1.9+10.11.15`, mumble `1.0.0+v1.6.870-0`, bluesky-pds `0.3.0+v0.4.219`, gitea `3.6.0+1.24.2-rootless`.
|
||
|
||
- **discourse** — EXPECT install/backup/restore/custom pass, upgrade fail on `test_head_runs_official_image_not_bitnamilegacy` + `test_sidekiq_service_dropped_by_head`. Confirm the overlay mismatch statically: `git -C ~/.abra/recipes/discourse show 0.8.1+3.5.0:compose.yml | grep -A1 ' app:'` and `... show main:compose.yml` both = `bitnamilegacy/discourse:3.5.0`; `grep -c 'sidekiq:'` = 1 in both. So the test's `discourse/discourse:3.5.3`/no-sidekiq expectation exists nowhere upstream.
|
||
- **mattermost-lts** — EXPECT restore fail `relation "ci_marker" does not exist`. Confirm root cause statically: `git -C ~/.abra/recipes/mattermost-lts show 2.1.9+10.11.15:compose.yml | grep backupbot` shows pre-hook + `backup.path` but NO `restore.post-hook`; immich `git -C ~/.abra/recipes/immich show <latest>:compose.yml | grep backupbot` shows `restore.post-hook: /pg_backup.sh restore`.
|
||
- **mumble** — EXPECT all tiers green (run 2–3× to confirm reproducibly green isolated). Canonical written green: `cat /var/lib/ci-warm/mumble/canonical.json`.
|
||
- **bluesky-pds** — EXPECT cold green, WC5 promote `!! WC5 promote failed … warm-bluesky-pds … last status 0`. While the warm stack is up, confirm root cause: caddy logs `dial tcp 10.10.0.X:3000: connect: connection refused` for `app:3000/tls-check`; `docker exec <caddy> getent hosts app` returns proxy IPs (10.10.0.X), the app's real internal IP is 10.0.3.x; `docker network inspect proxy | grep _app` shows many stacks aliasing `app`. (Tear down the orphaned warm-bluesky-pds stack + volumes after.)
|
||
- **gitea** — REQUIRES idle canonical first: if warm-gitea is deployed, `docker stack rm warm-gitea_ci_commoninternet_net` (retains data+config volumes) so the advance reattaches from idle. EXPECT cold green, warm advance crash-loop with container log `LoadCommonSettings() [F] error saving JWT Secret … "/etc/gitea/app.ini": read-only file system`. Restore: leave warm-gitea undeployed (idle 3.5.3, volumes retained) — registry stays `3.5.3+1.24.2-rootless`.
|
||
- **keycloak** — no run. Code-verify: `canonical.canonical_domain('keycloak')` → `warm.stable_domain('keycloak')` → `warm-keycloak.ci.commoninternet.net`; `warm.WARM_DOMAINS['keycloak']` == same string (runner/harness/canonical.py:42-44, warm.py:27-29,44-48). Live keycloak 200 on `/realms/master`.
|
||
|
||
### Node state left clean
|
||
All isolation runs torn down; orphaned warm-bluesky-pds stack+volumes removed; warm-gitea restored to idle 3.5.3 (volumes retained, registry unchanged); only live warm-keycloak deployed (healthy). No `run_recipe_ci.py` processes.
|
||
|
||
## M1 — PASS @ 2026-06-18T01:18Z (REVIEW-redfix.md; all 6 classifications cold-verified CORRECT by Adversary's own isolation re-runs). No VETO. Cleared to M2.
|
||
|
||
## Phase: M2 — FIX + verify all six (IN PROGRESS)
|
||
|
||
Fix designs locked in BACKLOG-redfix.md. Recipe PRs (mattermost-lts/bluesky/gitea) on git.autonomic.zone
|
||
mirrors via the recipe mirror+PR flow, verified `!testme` (NEVER merge). Harness fixes (keycloak/mumble)
|
||
on a cc-ci branch, verified via the harness. discourse: overlay-scope decision. Node now free for my
|
||
deploys (Adversary done with M1).
|
||
|
||
### M2 fix tracker (updated 2026-06-18T05:53Z — ALL VERIFIED)
|
||
|
||
| Recipe | Class | Fix | PR/branch + ref | Status |
|
||
|---|---|---|---|---|
|
||
| mattermost-lts | recipe defect | pg_backup.sh + `backupbot.restore.post-hook` (immich pattern) | mirror PR #1 `ci/pg-restore` @4ca7f418 | **VERIFIED** — !testme run #901 ALL tiers green incl `test_restore_returns_state` |
|
||
| discourse | stale cc-ci overlay | recipe: bitnamilegacy->official discourse image migration + drop orphaned image-less sidekiq from compose.smtpauth.yml (F-redfix-1) | mirror PR #4 `discourse-official-image` @9ff5e19 | **VERIFIED** — own cold run `/tmp/redfix-discourse-m2verify.log` **level=5 of 5** (all tiers + lint R011 PASS); F-redfix-1 regression fixed |
|
||
| keycloak | harness defect | collision-free `canonical_domain` (`warm-canon-<r>` for WARM_DOMAINS recipes) + enroll | cc-ci branch `redfix-m2-harness` @61211db | **VERIFIED** — branch-checkout run promotes at warm-canon-keycloak; live warm-keycloak 200 throughout |
|
||
| mumble | load/timing flake | harness: handshake readiness budget 60s->180s | cc-ci branch `redfix-m2-harness` @07fc6d4 | **VERIFIED** — branch-checkout run all tiers green incl handshake; budget active+non-regressing |
|
||
| gitea | recipe defect | app.ini->staging `/etc/gitea/app.ini.init` + docker-setup seed-on-EMPTY + DOCKER_SETUP_SH_VERSION v3 | mirror PR #2 `ci/app-ini-writable` @a0f2db8 | **VERIFIED** (direct chaos-deploy; promote merge-gated — see below) |
|
||
| bluesky-pds | recipe defect (routing) | caddy `{$APP_HOST}=${STACK_NAME}_app` (operator: NO rename) + CADDYFILE_VERSION v2 | mirror PR #4 `ci/warm-routing-alias` @4987ba9 | **VERIFIED** (direct chaos-deploy; promote merge-gated — see below) |
|
||
|
||
cc-ci-side change verification: run from a checkout of `redfix-m2-harness` (CCCI_REPO=<checkout>);
|
||
never touches /etc/cc-ci main. `redfix-m2-harness` is now mumble+keycloak ONLY (bluesky needs no
|
||
cc-ci change with the ${STACK_NAME}_app approach; the rename's exec-ref commit b96b8a4 was dropped).
|
||
|
||
## Gate: M2 — RE-CLAIMED, awaiting Adversary (2026-06-18T06:55Z; orig claim 05:53Z)
|
||
|
||
**Re-claim delta (addresses Adversary M2 FAIL @06:42Z — finding F-redfix-1).** The first M2 verdict was
|
||
FAIL on discourse ONLY (other 5 PASS, do-not-redo). F-redfix-1: the official-image migration dropped
|
||
`sidekiq` from compose.yml but left a dangling image-less `sidekiq:` block in `compose.smtpauth.yml` →
|
||
L5 lint R011 fail (run level=4) + broken SMTP-auth deploy. **FIXED** in PR #4 `discourse-official-image`
|
||
@**9ff5e19** (force-pushed onto @53ba0910): dropped the orphaned `sidekiq:` block; the `app:` override
|
||
already carries `DISCOURSE_SMTP_PASSWORD_FILE` + `smtp_password` secret (sidekiq is internal to the
|
||
official image), so no SMTP coverage lost. `grep sidekiq compose*.yml` = 0.
|
||
**VERIFIED two ways:** (1) the Adversary's exact lint.py repro flow at 9ff5e19 → **R011 ✅**; (2) my own
|
||
full cold run `/tmp/redfix-discourse-m2verify.log` → `RUN SUMMARY ... level=5 of 5`, all tiers pass
|
||
(install/upgrade/backup/restore/custom), `lint rung: pass`. Node clean: no discourse stack, NO discourse
|
||
canonical (untagged migrated head correctly does not promote — should_promote tagged-gate), recipe reset
|
||
to published tag 0.8.1+3.5.0. The other 5 fixes are unchanged since their Adversary PASS (keycloak,
|
||
mumble, gitea, bluesky-pds, mattermost-lts) — no re-run needed.
|
||
|
||
Adversary cold-verify for discourse: clone discourse @9ff5e19, run `RECIPE=discourse CCCI_SKIP_FETCH=1
|
||
… run_recipe_ci.py` → EXPECT level=5 of 5 (lint R011 ✅, all tiers pass, both upgrade-overlay tests
|
||
`test_head_runs_official_image_not_bitnamilegacy` + `test_sidekiq_service_dropped_by_head` pass); OR the
|
||
lint-only repro in F-redfix-1 → R011 ✅. `grep -c sidekiq ~/.abra/recipes/discourse/compose*.yml` @9ff5e19 = 0.
|
||
|
||
---
|
||
## Gate: M2 — original claim (2026-06-18T05:53Z)
|
||
|
||
**WHAT (M2 DoD).** All six canon-sweep failures FIXED — each via a recipe PR or a harness improvement —
|
||
and verified green. No recipe left as a standing exception. Nothing merged (operator merges). Per recipe:
|
||
|
||
- **mattermost-lts** (recipe PR #1) — added `pg_backup.sh` + postgres `backupbot.restore.post-hook` so
|
||
the logical dump round-trips on restore.
|
||
- **discourse** (recipe PR #4) — migrated the head off deprecated `bitnamilegacy` to the official
|
||
`discourse/discourse` image so the stale PR-faithfulness overlay (`test_head_runs_official_image…`,
|
||
`test_sidekiq_service_dropped…`) passes on the migrated head (NOT a test-weakening).
|
||
- **keycloak** (harness branch) — `canonical_domain` returns a collision-free `warm-canon-<r>` for
|
||
recipes in `warm.WARM_DOMAINS` (live-warm OIDC providers); keycloak enrolled (WARM_CANONICAL=True).
|
||
- **mumble** (harness branch) — handshake readiness budget widened 60s->180s (load-flake stabilization).
|
||
- **gitea** (recipe PR #2) — app.ini is now seeded into the WRITABLE `/etc/gitea` volume by
|
||
docker-setup (`if [ ! -s /etc/gitea/app.ini ]`, seed-on-EMPTY) from the read-only staging config
|
||
`app.ini.init`; `DOCKER_SETUP_SH_VERSION` v1->v3 forces the new docker-setup to re-mount. Gitea
|
||
1.24.2 can then persist its JWT secret (the M1 read-only-app.ini crash is gone).
|
||
- **bluesky-pds** (recipe PR #4) — caddy resolves its OWN app via the fully-qualified swarm name
|
||
`${STACK_NAME}_app` (caddy `{$APP_HOST}` env, set in the caddy service) instead of bare `app`, which
|
||
collided with other stacks' `app` aliases on the shared `proxy` net. CADDYFILE_VERSION v1->v2.
|
||
|
||
**HOW + EXPECTED + WHERE (Adversary cold-verify, one recipe at a time, no concurrent load):**
|
||
|
||
- **mattermost-lts** — read-only artifact: `/var/lib/cc-ci-runs/901/` on cc-ci — all tiers pass,
|
||
`junit/restore__cc-ci__test_restore.xml` testsuite failures=0, `test_restore_returns_state` pass.
|
||
OR re-run !testme on PR #1 @4ca7f418. EXPECT restore green.
|
||
- **discourse** — !testme on PR #4 @53ba0910 (run #849 green) OR run from a checkout of the migrated
|
||
head: EXPECT install/backup/restore/custom + upgrade overlay all pass (head now official image).
|
||
- **keycloak** — from a `redfix-m2-harness` @61211db checkout (CCCI_REPO=<checkout>), run
|
||
`RECIPE=keycloak CCCI_SKIP_FETCH=1 ... run_recipe_ci.py`. EXPECT all cold tiers pass + WC5 promote
|
||
succeeds at domain `warm-canon-keycloak.ci.commoninternet.net` (NOT warm-keycloak); live
|
||
`warm-keycloak.ci.commoninternet.net/realms/master` stays 200 throughout. Code: `canonical.py`
|
||
canonical_domain returns warm-canon-<r> for r in warm.WARM_DOMAINS.
|
||
- **mumble** — from `redfix-m2-harness` @07fc6d4 checkout, run `RECIPE=mumble CCCI_SKIP_FETCH=1 …`.
|
||
EXPECT all 5 tiers green incl `custom/test_protocol_handshake.py::test_handshake_completes_with_
|
||
channel_presence`; handshake budget = 36 attempts / 180s (was 60s). (Load-flake is not
|
||
deterministically reproducible; this verifies the stabilization is applied, sound, non-weakening.)
|
||
- **gitea** (recipe PR #2 @a0f2db8 on mirror branch `ci/app-ini-writable`) — DIRECT chaos-deploy proof
|
||
(the harness WC5 promote is merge-gated, see NOTE). With the idle 3.5.3 canonical present:
|
||
`cd ~/.abra/recipes/gitea && git checkout -f a0f2db8` then chaos-deploy onto the retained canonical
|
||
volumes (0-byte app.ini = genuine pre-fix 3.5.3 state):
|
||
`abra app deploy warm-gitea.ci.commoninternet.net -C -o -n`. EXPECT: service 1/1; the config volume's
|
||
`app.ini` seeded 0->~1862 bytes (`INSTALL_LOCK = true`); `/api/v1/version` -> 200 {"version":"1.24.2"}
|
||
and `/api/healthz` -> 200 (curl inside the app container); retained 3.5.3 data adopted (data dirs
|
||
dated 2026-06-17T08:39); ZERO `read-only file system` crashes in `docker service logs` (M1 crashed
|
||
here). Evidence: `/tmp/redfix-gitea-m2-directproof.log` on cc-ci. Teardown: `abra app undeploy … -n`,
|
||
truncate the volume app.ini to 0 (restore pre-fix state). canonical.json stays 3.5.3 idle e6a1cc79.
|
||
- **bluesky-pds** (recipe PR #4 @4987ba9 on mirror branch `ci/warm-routing-alias`) — DIRECT chaos-deploy
|
||
proof (warm-promote is the only failing path; merge-gated). `git checkout -f 4987ba9`; generate
|
||
secrets (`abra app secret generate warm-bluesky-pds.ci.commoninternet.net --all -m -C -o -n`) + insert
|
||
a PLC rotation key (tests/bluesky-pds/install_steps.sh logic: 32-byte hex into pds_plc_rotation_key
|
||
v1); **re-checkout 4987ba9 AFTER secret ops** (abra secret insert force-fetches+reverts the checkout);
|
||
`abra app deploy warm-bluesky-pds.ci.commoninternet.net -C -o -n` (EXPECT `caddyfile: v1 -> v2`,
|
||
NEW DEPLOYMENT 4987ba9). EXPECT: app+caddy 1/1; inside caddy `getent hosts
|
||
warm-bluesky-pds_ci_commoninternet_net_app` -> a 10.0.x.x INTERNAL ip (own stack) while
|
||
`getent hosts app` -> a 10.10.x.x proxy ip (foreign, the M1 collision); caddy log "certificate
|
||
obtained successfully" with 0 "connection refused"; external `curl https://warm-bluesky-pds.ci.
|
||
commoninternet.net/xrpc/_health` -> **200** {"version":"0.4.219"} (M1 was 000). Evidence:
|
||
`/tmp/redfix-bluesky-m2-directproof.log`. Teardown: undeploy + remove volumes (caddy_data, pds_data)
|
||
+ secrets (no canonical, matching M1).
|
||
|
||
**NOTE — gitea & bluesky end-to-end canonical-promote is OPERATOR-MERGE-GATED (not a shrug).** The
|
||
harness WC5 promote does a recipe_checkout(published-tag)+non-chaos deploy, and BOTH run_recipe_ci.py:373
|
||
AND abra force-fetch `refs/tags/*` from upstream (abra.py:135 documents this), so any local move of the
|
||
release tag to the fix commit is reverted to the PUBLISHED commit. The published 3.6.0 / 0.3.0 tags do
|
||
NOT yet carry the fix (PR not merged — operator merges, per phase guardrail), so pre-merge the promote
|
||
necessarily deploys the unfixed published release. Confirmed empirically: a full gitea harness run's WC5
|
||
promote deployed 357926f and crash-looped exactly like M1. The DIRECT chaos-deploy (chaos = deploy the
|
||
working-tree checkout = the PR fix) is therefore the MAXIMAL + faithful pre-merge proof — it reproduces
|
||
the EXACT M1 failing scenario (gitea: the retained canonical volumes; bluesky: warm-bluesky-pds on the
|
||
shared proxy) and shows the fix resolves it. End-to-end canonical advance follows automatically once the
|
||
operator merges PR #2 / #4 and the release tag carries the fix. This is NOT a standing exception — the
|
||
defect is fixed + proven; only the registry-advance awaits the operator's merge (the phase's own
|
||
"nothing merged" constraint).
|
||
|
||
**WHERE (refs).** Recipe PRs on `git.autonomic.zone/recipe-maintainers/<recipe>`: mattermost-lts
|
||
`ci/pg-restore`@4ca7f418, discourse `discourse-official-image`@53ba0910, gitea `ci/app-ini-writable`
|
||
@a0f2db8, bluesky-pds `ci/warm-routing-alias`@4987ba9. cc-ci harness branch
|
||
`redfix-m2-harness`@07fc6d4 (keycloak 61211db + mumble 07fc6d4). Reasoning/dead-ends in
|
||
JOURNAL-redfix.md. Node left clean (only infra + live warm-keycloak 200; gitea idle 3.5.3 volumes
|
||
retained, canonical e6a1cc79 unchanged; no bluesky/test stacks/volumes/secrets; no run procs).
|
||
|
||
## Gate: M1 — PASS (above).
|
||
|
||
**WHAT (M1 DoD).** All six canon-sweep failures investigated in ISOLATION (one recipe at a time, no
|
||
concurrent sweep load), root-caused with first-hand evidence, and classified (flake vs genuine; recipe
|
||
vs test vs warm-machinery vs load) — see the **M1 results table** + **HOW the Adversary cold-verifies**
|
||
sections above. Summary: discourse = stale cc-ci overlay test (canon timeout/FATA root-cause was
|
||
wrong); mattermost-lts = genuine recipe defect (no `backupbot.restore.post-hook`); mumble = load/timing
|
||
FLAKE (2× isolation green); bluesky-pds = genuine routing defect (caddy↔app `app`-alias collision on
|
||
shared proxy); gitea = genuine recipe defect (read-only app.ini config mount + 3.6.0 JWT save);
|
||
keycloak = harness warm-domain namespace collision. NO "probably a flake" — every classification has
|
||
an isolation re-run or code proof.
|
||
|
||
**HOW + EXPECTED + WHERE.** Per-recipe cold-verify commands, expected outputs, and evidence paths are
|
||
in the two sections above ("M1 results table" and "HOW the Adversary cold-verifies each classification").
|
||
Evidence logs on cc-ci: `/tmp/redfix-{discourse,mattermost-lts,mumble,mumble2,bluesky-pds,gitea2}.log`.
|
||
Reasoning/dead-ends in JOURNAL-redfix.md. Node left clean (see "Node state left clean" above).
|
||
|
||
## Untracked file `main.go` — NOT produced by this phase (operator: unexplained)
|
||
|
||
**WHAT.** An untracked `main.go` (281 bytes, a Go "Hello, World!" `net/http` server on `:8080`) sits in the
|
||
repo root. It is not in any commit on any ref, is not gitignored, and is unrelated to every redfix DoD item.
|
||
No `go.mod` accompanies it. I did not create it and have not committed, modified, or deleted it.
|
||
|
||
**WHERE.** `/srv/cc-ci-orch/cc-ci/main.go` — **one file in one clone.**
|
||
|
||
**RETRACTED (2026-07-09, Adversary A-redfix-3):** an earlier version of this entry claimed the file was
|
||
"present in **both** clones … i.e. written by something outside git." **That evidence is void.** `/srv/cc-ci`
|
||
is a **symlink to `/srv/cc-ci-orch`**, so `/srv/cc-ci/cc-ci/main.go` and `/srv/cc-ci-orch/cc-ci/main.go` are
|
||
the **same inode** — one file seen twice, not two clones. The "a git operation cannot leave the same untracked
|
||
file in two clones" inference therefore carries **no information** about the file's origin, which remains
|
||
**unexplained**. Independently re-derived: `ls -la /srv/` shows `cc-ci -> /srv/cc-ci-orch`, and
|
||
`stat -c '%i %h %n'` on both paths prints the same inode `3254604` with `links=1`.
|
||
|
||
**HOW to verify.** `git log --all --oneline -- main.go` → empty (in no commit).
|
||
`git status --short` → `?? main.go`.
|
||
`ls -la /srv/ | grep cc-ci` → `cc-ci -> /srv/cc-ci-orch` (the symlink that voids the "both clones" claim).
|
||
`stat -c '%i %h %n' /srv/cc-ci/cc-ci/main.go /srv/cc-ci-orch/cc-ci/main.go` → same inode, `links=1`.
|
||
|
||
**Risk: inert** (Adversary-confirmed). 281-byte hello-world `net/http` listener; no `go.mod`; `go` is **not
|
||
installed**; **nothing listening on `:8080`**; in no commit on any ref; unreferenced by any tracked file.
|
||
|
||
**EXPECTED.** Left in place, untracked. Per guardrails I do not delete or commit files I did not create;
|
||
flagging for the operator rather than acting. It affects no gate, no DoD item, and no verdict.
|
||
|
||
## Blocked
|
||
|
||
(none)
|