d55ba23a29
Some checks failed
continuous-integration/drone/push Build is failing
A bare '## VETO —' heading still matched a grep for a standing veto even though it was cleared at 00:18Z, and the Builder's DONE rests on 'no standing VETO'. Annotated the original heading in place (history preserved) and made '## VETO CLEARED' a heading rather than bold text. No verdict changes: M2 PASS stands, F-redfix-4 stays CLOSED.
654 lines
55 KiB
Markdown
654 lines
55 KiB
Markdown
# REVIEW — phase `redfix` (Adversary)
|
||
|
||
Phase SSOT: `/srv/cc-ci/cc-ci-plan/plan-phase-redfix-canon-sweep-failures.md`
|
||
Mission: investigate every canon-sweep failure (discourse, mattermost-lts, mumble, bluesky-pds,
|
||
gitea, keycloak), isolate → root-cause → classify (flake vs genuine; recipe vs test vs
|
||
warm-machinery vs load) → FIX each via a recipe PR or harness improvement → verify green.
|
||
No standing exceptions. Nothing merged.
|
||
|
||
Gates:
|
||
- **M1** — all six investigated in isolation, classified with evidence. Adversary cold-verifies:
|
||
claimed flake = reproducibly green in isolation (and red under load); claimed recipe defect =
|
||
genuinely the recipe (not a stale test / harness artifact); claimed warm-machinery bug = in cc-ci.
|
||
- **M2** — all six FIXED + verified green (recipe PR via `!testme`; harness/cc-ci PR via the harness;
|
||
flake-stabilization green under load). All six promote/pass. No standing exception. Nothing merged.
|
||
|
||
DONE = Builder writes `## DONE` only after M1+M2 fresh Adversary PASS here.
|
||
|
||
---
|
||
|
||
## Verdicts
|
||
|
||
### M1 — investigate + isolate + classify: **PASS** @ 2026-06-18T01:18Z
|
||
|
||
Gate claim: `claim(redfix-M1)` commit `0a06c41` (@00:25Z). Verified from a COLD START on cc-ci with my
|
||
OWN isolation re-runs (one recipe at a time, no concurrent load) — NOT the Builder's logs. Isolation
|
||
discipline honored: verdict formed from the phase plan (SSOT), the recipe code / git history, the
|
||
verification info in STATUS, and my own cold acceptance runs; I did NOT read JOURNAL-redfix.md before
|
||
writing this verdict.
|
||
|
||
All six classifications are CORRECT. Evidence per recipe (full detail in the verification log below):
|
||
|
||
| Recipe | My independent reproduction | Classification — verified |
|
||
|---|---|---|
|
||
| **discourse** | my isolation run `/tmp/adv-discourse.log`: install/backup/restore/custom PASS, upgrade FAIL on the 2 PR-faithfulness overlay asserts; **converged in minutes, no FATA/rc=142/wedge** | **stale/PR-specific cc-ci OVERLAY test** (canon "timeout" root-cause was WRONG — confirmed). Recipe deploys+serves fine. ✔ |
|
||
| **mattermost-lts** | my isolation run `/tmp/adv-mattermost.log`: **restore FAIL deterministically** (`relation "ci_marker" does not exist`, 91s, isolated) | **genuine RECIPE defect** — no `backupbot.restore.post-hook`; NOT the canon "loaded-node race." ✔ |
|
||
| **mumble** | my isolation run `/tmp/adv-mumble.log`: ALL 5 tiers GREEN incl `test_handshake_completes_with_channel_presence`; promote OK | **load/timing FLAKE** — green in isolation (a recipe defect would red deterministically; it didn't). ✔ |
|
||
| **bluesky-pds** | my isolation run `/tmp/adv-bluesky.log` + live caddy diag: cold GREEN, warm promote **000 deterministic**; `getent app`→10.10.0.4 (foreign proxy), own app 10.0.5.6 never resolved; caddy log cycles `dial 10.10.0.{4..12}:3000 refused` | **genuine recipe ROUTING defect** (bare `app` + caddy on shared `proxy`), NOT cc-ci promote-machinery (it correctly refused to promote), NOT flake. (Reverses the plan's "warm-machinery" prior — confirmed against it.) ✔ |
|
||
| **gitea** | my isolation run `/tmp/adv-gitea.log` + container crash log: cold GREEN, warm advance crash-loops 0/1; `LoadCommonSettings() [F] … error saving JWT Secret … "/etc/gitea/app.ini": read-only file system`; canonical correctly stayed 3.5.3 (promote timed out, refused) | **genuine RECIPE defect** (3.6.0 JWT save vs read-only app.ini docker-config mount; `/etc/gitea` is a writable volume but the app.ini file is the RO config). ✔ |
|
||
| **keycloak** | code-verified: `canonical.canonical_domain('keycloak')`→`warm.stable_domain`→`warm-keycloak.ci.commoninternet.net` == `warm.WARM_DOMAINS['keycloak']` (warm.py:47 documents the equality); live keycloak 200 on `/realms/master` | **HARNESS defect** (data-warm canonical domain collides with the live-warm OIDC provider; no collision-free namespace). ✔ |
|
||
|
||
No defects in the classification work. No VETO. Node verified clean before AND after my runs (only infra
|
||
+ live warm-keycloak; gitea restored to undeployed idle 3.5.3, volumes retained, canonical commit
|
||
`e6a1cc79` unchanged; warm-keycloak healthy throughout). **M1 PASS — Builder cleared to proceed to M2.**
|
||
(M2 will re-verify each FIX green; this PASS is for the investigation/classification gate only.)
|
||
|
||
_(prior placeholder removed)_
|
||
|
||
## Adversary verification log
|
||
|
||
- 2026-06-17T23:18Z — Phase redfix opened. Refreshed phase plan + plan.md §6.1. Cold access to cc-ci
|
||
confirmed (`ssh cc-ci`: host `nixos`, uptime 4d, `systemctl --failed` empty, load ~0.8). No Builder
|
||
state files (`STATUS/BACKLOG/JOURNAL-redfix.md`) yet; no gate claimed. Idling for the first claim.
|
||
- 2026-06-18T00:10Z — Non-contending pre-staging (M1 NOT yet claimed; Builder mid-investigation:
|
||
gitea isolation running, keycloak pending). Stayed OFF the swarm to avoid contaminating the
|
||
Builder's isolation runs. Independently corroborated two deterministic static claims via pure
|
||
code reads on cc-ci (no deploys):
|
||
* **mattermost-lts** (recipe @ `2.1.9+10.11.15`): postgres svc has `backupbot.backup.pre-hook`
|
||
(pg_dump → /var/lib/postgresql/data/postgres-backup.sql), `backup.post-hook` (rm dump),
|
||
`backup.path=/var/lib/postgresql/data/` (hot live PGDATA) — and **NO `backupbot.restore.post-hook`**.
|
||
immich (passes) uses dump-only `backup.volumes.postgres.path: backup.sql` + `restore.post-hook:
|
||
/pg_backup.sh restore`. Corroborates "genuine recipe defect — no restore round-trip." ✔ pre-staged.
|
||
* **discourse** (recipe @ `0.8.1+3.5.0` = `bitnamilegacy/discourse:3.5.0` + sidekiq): overlay
|
||
`tests/discourse/test_upgrade.py` is a phase-prevb PR-faithfulness test asserting app image ==
|
||
official `discourse/discourse:3.5.3` AND sidekiq dropped — only true on an unreleased PR head, not
|
||
the latest release the canon sweep deploys. So it red-by-construction in the sweep. Corroborates
|
||
"stale/PR-specific overlay test, not flake/timeout/recipe-deploy." ✔ pre-staged.
|
||
* STILL OWED before any M1 PASS: my OWN cold isolation run of discourse to confirm the
|
||
re-classification from the original canon hypothesis ("cold-deploy timeout, ~51-min wedge") to
|
||
"deploys+serves fine, only the overlay test reds." Will run when M1 is claimed and the swarm is
|
||
free (Builder not deploying). Same for bluesky app-alias collision (needs live caddy/getent diag).
|
||
These are NOT verdicts — formal M1 PASS/FAIL awaits the Builder's gate claim.
|
||
- 2026-06-18T00:25Z — **M1 CLAIMED** (commit 0a06c41). Node verified idle/clean before any run
|
||
(only infra + live warm-keycloak; no bluesky/test stacks; no run_recipe_ci; load 0.03; gitea idle
|
||
3.5.3) — Builder "node clean" claim ✔. Began my own COLD isolation re-runs (one at a time, no
|
||
concurrent load), swarm confirmed free.
|
||
- 2026-06-18T00:29Z — **bluesky-pds CONFIRMED by my own reproduction** (`/tmp/adv-bluesky.log`,
|
||
tag 0.3.0+v0.4.219, RECIPE=bluesky-pds CCCI_SKIP_FETCH=1). Cold lifecycle GREEN (install/backup/
|
||
restore/custom=pass, upgrade=skip) — reproduced. WC5 promote → unhealthy, 000. DECISIVE live diag
|
||
inside the warm caddy container (60326521a2ac, nets: proxy=10.10.52.13 + internal=10.0.5.3):
|
||
* `getent hosts app` → **10.10.0.4** (a *proxy*-net foreign endpoint) — NOT bluesky's own app.
|
||
* bluesky's OWN app is at internal **10.0.5.6** (real target), never resolved.
|
||
* caddy TLS log cycles `dial tcp 10.10.0.{4,5,6,8,10,11,12}:3000: connect: connection refused`
|
||
on `ask http://app:3000/tls-check` → on-demand cert denied → TLS fails → /xrpc/_health = 000.
|
||
Verdict basis: NOT a flake (deterministic, every retry refused); NOT promote-machinery (the probe
|
||
correctly refuses an unhealthy endpoint, no false promote); **genuine recipe routing defect** —
|
||
recipe names its svc `app` + puts caddy on the shared multi-tenant `proxy` net + Caddyfile uses bare
|
||
`app`, so docker DNS resolves `app` to OTHER stacks' apps. Builder's classification (recipe defect,
|
||
reverses the plan's "cc-ci warm-machinery" prior) is CORRECT. Sharper than Builder's note (my run's
|
||
internal IP 10.0.5.6 vs their 10.0.3.3 — same mechanism, different deploy). Letting run finish + will
|
||
tear down the orphan warm-bluesky stack. [interim — full M1 verdict batched after mumble+discourse.]
|
||
- 2026-06-18T00:38Z — bluesky run finished; promote log `!! WC5 promote failed (non-fatal; known-good
|
||
unchanged) … last status 0` — **machinery correctly refused to write canonical** (seals "not
|
||
promote-machinery"). Cleaned up: `docker stack rm warm-bluesky-pds…` + removed both volumes
|
||
(caddy_data, pds_data). Node verified clean of bluesky.
|
||
- 2026-06-18T00:44Z — **mumble CONFIRMED by my own isolation run** (`/tmp/adv-mumble.log`, tag
|
||
1.0.0+v1.6.870-0). ALL 5 tiers GREEN: install/upgrade/backup/restore/custom = pass. The exact
|
||
canon-sweep failure `tests/mumble/custom/test_protocol_handshake.py::test_handshake_completes_with_
|
||
channel_presence` **PASSED** in isolation. WC5 promote SUCCEEDED (canonical advanced to known-good
|
||
1.0.0+v1.6.870-0, idle, volume retained). A recipe defect would fail deterministically in isolation
|
||
(cf. mattermost restore) — mumble passing cleanly confirms **load/timing FLAKE**, not a recipe bug.
|
||
(My 1 isolation green + Builder's 2× = 3 isolation greens / 0 isolation reds vs 1 canon red-under-load
|
||
— consistent flake signature.) Builder's classification CORRECT.
|
||
- 2026-06-18T00:53Z — **discourse CONFIRMED by my own isolation run** (`/tmp/adv-discourse.log`, tag
|
||
0.8.1+3.5.0). Tiers: **install pass / upgrade FAIL / backup pass / restore pass / custom pass** —
|
||
exactly the Builder's claim. Deploy **converged in minutes; NO FATA, NO rc=142/143, NO ~51-min
|
||
wedge** → the original canon "cold-deploy timeout" hypothesis did NOT reproduce in isolation (Builder
|
||
reclassification CORRECT). Upgrade failed on the two PR-faithfulness overlay assertions:
|
||
`test_head_runs_official_image_not_bitnamilegacy` (deployed image = `bitnamilegacy/discourse:3.5.0@
|
||
sha256:db7e...`, the release's own image) and `test_sidekiq_service_dropped_by_head` (services =
|
||
`['app','db','redis','sidekiq']`). The overlay demands official `discourse/discourse:3.5.3` + no
|
||
sidekiq — an unreleased PR migration in NO release tag and NOT in main (verified earlier: tag AND
|
||
main both `bitnamilegacy:3.5.0`+sidekiq). AssertionError self-documents "the prevb bug." So the
|
||
recipe DEPLOYS+SERVES fine; only the stale/PR-specific overlay reds by construction in the canonical
|
||
sweep. **stale cc-ci OVERLAY test**, not flake/timeout/recipe-deploy/warm-machinery. Builder CORRECT.
|
||
- 2026-06-18T01:02Z — **mattermost-lts CONFIRMED by my own isolation run** (`/tmp/adv-mattermost.log`,
|
||
tag 2.1.9+10.11.15). Tiers: install pass / upgrade pass / backup pass / **restore FAIL** / custom
|
||
pass — exactly Builder's claim. The overlay `tests/mattermost-lts/test_restore.py::
|
||
test_restore_returns_state` FAILED with the EXACT `RuntimeError: docker exec … postgres failed
|
||
(rc=1): ERROR: relation "ci_marker" does not exist`. **Deterministic in isolation** (91s, no
|
||
concurrent load) → NOT the canon "loaded-node db-cycle race." Note: generic `test_restore_healthy`
|
||
PASSED (app returns healthy) but the STATE round-trip failed — the seeded marker is gone after
|
||
restore. Mechanism matches the static finding: backup dumps + backs up hot PGDATA but has NO
|
||
`backupbot.restore.post-hook` to replay the dump → postgres logical data never round-trips. **genuine
|
||
RECIPE defect**, not a flake/load-race/stale-test. Builder's classification CORRECT.
|
||
- 2026-06-18T01:09Z — **gitea CONFIRMED by my own isolation run + container crash log**
|
||
(`/tmp/adv-gitea.log`, tag 3.6.0+1.24.2-rootless). Cold lifecycle all 5 tiers GREEN (incl fresh
|
||
3.5.3→3.6.0 upgrade tier). WC5 advance (reattach idle 3.5.3 volumes with 3.6.0 image) → warm-gitea
|
||
app crash-loops 0/1. Container log (every task, e.g. .8zd4952…): `setting.go:105:LoadCommonSettings()
|
||
[F] Unable to load settings from config: error saving JWT Secret for custom config: failed to save
|
||
"/etc/gitea/app.ini": open /etc/gitea/app.ini: read-only file system`. Mount nuance CONFIRMED:
|
||
`/etc/gitea` is a writable VOLUME (RW=true) but app.ini is a docker CONFIG overlaying that path
|
||
read-only → gitea can write the dir but NOT the app.ini file. **genuine RECIPE defect** (3.6.0 JWT
|
||
save vs read-only app.ini config mount). Cold passes (fresh render, no runtime save). Builder's
|
||
classification + proposed fix (render app.ini into the writable volume) CORRECT. Will verify
|
||
canonical stays 3.5.3 (promote refused) + restore warm-gitea to undeployed idle.
|
||
|
||
- 2026-06-18T02:15Z — **M2 interim corroboration (NOT a verdict — M2 not yet claimed).** Node cold-checked
|
||
idle (load 0.07, no run_recipe_ci/abra, only live warm-keycloak) — Builder between M2 fixes, so I stayed
|
||
OFF the swarm (no contending deploy). Non-contending read-only check of the one fix marked DONE
|
||
(mattermost-lts PR #1, ref `4ca7f4182d83`): cc-ci run **#901** artifacts on cc-ci
|
||
(`/var/lib/cc-ci-runs/901/`) confirm all tiers pass (install/upgrade/backup/restore/custom), rungs all
|
||
pass, `flags.clean_teardown=true`, `flags.no_secret_leak=true`, `WARM_CANONICAL=true`. The exact
|
||
M1-failing test now PASSES: `junit/restore__cc-ci__test_restore.xml` → testsuite
|
||
`failures="0" errors="0" skipped="0" tests="1"`, testcase `test_restore_returns_state`. This is a
|
||
read-only artifact check, NOT my own cold re-run — the formal M2 PASS will require my own cold
|
||
re-verification of all six fixes once the Builder claims M2. Pre-staged anchor only.
|
||
|
||
- 2026-06-18T04:12Z — **Idle break-it probe (NOT a verdict — M2 not yet claimed).** Cold-checked node
|
||
while Builder reworks bluesky+gitea (their journal: 4/6 verified, bluesky warm-verify structurally
|
||
blocked pre-merge, gitea needs rework). Stayed OFF the swarm. Observations: live
|
||
`warm-keycloak.ci.commoninternet.net/realms/master` = **200** (live shared SSO undisturbed by the
|
||
keycloak harness fix + its verify run — the keycloak DoD's hard constraint holds). Deployed stacks =
|
||
infra + live warm-keycloak + a `warm-gitea` (Builder's active rework; app `/api/v1/version`=404 =
|
||
wizard mode, consistent with their "gitea fix v1 broke 3.5.3→3.6.0 transition"). No orphan
|
||
test/bluesky stacks, no `run_recipe_ci` procs, load 0.44. **Critical break-it check PASSED: gitea
|
||
canonical is UNCHANGED** — `/var/lib/ci-warm/gitea/canonical.json` still `3.5.3+1.24.2-rootless`,
|
||
commit `e6a1cc79`, status `idle`, ts `20260617T083930Z` (identical to M1). The Builder's broken gitea
|
||
fix attempts did NOT falsely promote 3.6.0 to canonical. Idling for the M2 gate claim.
|
||
|
||
---
|
||
## M2 gate verification (CLAIMED 2026-06-18T05:53Z) — component re-runs in progress
|
||
|
||
Verifying all 6 fixes from a COLD START via my own independent harness checkout (`/tmp/adv-m2` on cc-ci
|
||
@ origin/redfix-m2-harness b96b8a4 = keycloak 61211db + mumble 07fc6d4 + bluesky exec-into-pds b96b8a4)
|
||
and my own chaos-deploys. One recipe at a time, no concurrent load. Node idle at start (load 0.02, only
|
||
live warm-keycloak). Static code review of the harness branch first: canonical.py adds `warm-canon-<r>`
|
||
for r in `warm.WARM_DOMAINS` (ONLY keycloak — confirmed, so zero blast radius on the other 15
|
||
canonicals); mumble widens handshake budget 12->36 attempts (60s->180s) with the asserts UNCHANGED
|
||
(non-weakening); keycloak recipe_meta WARM_CANONICAL False->True. All three are genuine, not
|
||
test-disabling.
|
||
|
||
- 2026-06-18T06:08Z — **keycloak component VERIFIED (1/6)** by my OWN cold harness run
|
||
(`/tmp/adv-keycloak-m2.log`, RECIPE=keycloak from /tmp/adv-m2 @b96b8a4, recipe tag 10.8.0+26.6.3).
|
||
RUN SUMMARY: deploy-count=1, **all 5 cold tiers pass** (install/upgrade/backup/restore/custom incl
|
||
`custom/test_password_grant_token.py::test_password_grant_issues_valid_jwt`). **WC5 promote landed at
|
||
the COLLISION-FREE domain**: `/var/lib/ci-warm/keycloak/canonical.json` domain=
|
||
`warm-canon-keycloak.ci.commoninternet.net`, version 10.8.0+26.6.3, status idle, ts 20260618T060549Z
|
||
(THIS run). Promote genuinely DEPLOYED there — its own volumes exist (`warm-canon-keycloak_…_mariadb`,
|
||
`_providers`). **Hard invariant HOLDS — live shared SSO undisturbed**: live
|
||
`warm-keycloak_ci_commoninternet_net_app` up **4 days**, service last Updated **2026-06-13** (predates
|
||
my 06:04Z run by days → NOT bounced); `warm-keycloak.ci.commoninternet.net/realms/master` = **200**
|
||
before/during/after. The data-warm canonical (warm-canon-keycloak) and live-warm provider
|
||
(warm-keycloak) are fully separate deployments that never touched. Builder's keycloak fix CORRECT +
|
||
non-weakening; the §2.B de-enrollment is now structurally resolved. (1/6)
|
||
|
||
- 2026-06-18T06:15Z — **mumble component VERIFIED (2/6)** by my OWN cold harness run
|
||
(`/tmp/adv-mumble-m2.log`, RECIPE=mumble from /tmp/adv-m2, recipe tag 1.0.0+v1.6.870-0). RUN SUMMARY:
|
||
deploy-count=1, **all 5 cold tiers pass**. The stabilized custom test
|
||
`test_handshake_completes_with_channel_presence` **PASSED** (junit failures=0, time=10.3s). The
|
||
handshake completing in ~10s confirms M1's **load/timing-FLAKE** classification (fast in isolation,
|
||
nowhere near even the OLD 60s budget) and that the fix — widening 12->36 attempts (60s->180s) — is
|
||
pure headroom: the asserts are UNCHANGED, so a genuinely dead server still exhausts all 36 retries
|
||
and FAILs. **Non-weakening.** WC5 promote: `/var/lib/ci-warm/mumble/canonical.json` version
|
||
1.0.0+v1.6.870-0, idle, ts 20260618T061114Z (THIS run). Builder's mumble fix CORRECT. (2/6)
|
||
|
||
NOTE on branch state: I cloned /tmp/adv-m2 at tip `b96b8a4` just before the Builder force-reset
|
||
`redfix-m2-harness` to `07fc6d4` (dropping a bluesky exec-into-pds commit). Confirmed
|
||
`git diff 07fc6d4 b96b8a4` = ONLY `tests/bluesky-pds/_p4.py` + `test_account_and_post.py` (2 lines,
|
||
bluesky-only) → keycloak (61211db) and mumble (07fc6d4) code are BYTE-IDENTICAL between b96b8a4 and
|
||
the claimed tip 07fc6d4, so my keycloak+mumble PASSES hold at the claimed state. bluesky is verified
|
||
separately via recipe chaos-deploy (PR #4 @4987ba9, now recipe-PR-only per operator directive), so
|
||
the harness-checkout staleness does not touch it.
|
||
|
||
- 2026-06-18T06:18Z — **gitea component VERIFIED (3/6)** by my OWN direct chaos-deploy of recipe PR #2
|
||
@a0f2db8 onto the retained idle 3.5.3 canonical volumes (`/tmp/adv-gitea-m2.log`). This reproduces
|
||
the EXACT M1 warm-advance scenario. Two-sided proof: I verified the UNFIXED-crashes side first-hand
|
||
in M1 (`/tmp/adv-gitea.log`: read-only-file-system FATA at LoadCommonSettings). Now the FIX side:
|
||
* **Fix is genuine, not test-disabling** — compose.yml moves the read-only swarm config to
|
||
`/etc/gitea/app.ini.init`; docker-setup.sh.tmpl (v1->v3) seeds it into the WRITABLE `/etc/gitea`
|
||
volume **only when missing OR EMPTY** (`! -s`, handling the 0-byte placeholder the old direct-config
|
||
mount leaves); a non-empty app.ini (gitea's persisted state incl the JWT) is preserved.
|
||
* **Pre-state genuine pre-fix**: config-volume app.ini = **0 bytes**; retained 3.5.3 data (gitea.db
|
||
1347584 B dated 2026-06-17T08:39); canonical 3.5.3 idle e6a1cc79; stack not deployed.
|
||
* **Deploy result**: `deploy succeeded`, NEW DEPLOYMENT a0f2db88, docker_setup_sh v3. **service 1/1,
|
||
ZERO restarts** (task Running, no Error). **M1 read-only crash signature ABSENT** (grep of service
|
||
logs for `read-only file system`/`LoadCommonSettings`/`[F]` = empty). **app.ini seeded 0->1862 B**
|
||
with `[server] INSTALL_LOCK = true` (NOT wizard mode — the very bug that broke the Builder's v1
|
||
fix). `/api/v1/version` -> **200 {"version":"1.24.2"}**; `/api/healthz` -> **200**. Retained
|
||
gitea.db adopted in place (still 1347584 B @08:39, SQLite WAL active) — matches Builder's stated
|
||
adoption signal (data dirs @08:39). (Empty users/repos = minimal canonical install, not a
|
||
regression.)
|
||
* **Merge-gating is HONEST, not a shrug**: published 3.6.0 tag = commit 357926f (independently
|
||
confirmed) != fix commit a0f2db8, so a non-chaos WC5 promote deploys the unfixed release (the abra
|
||
force-fetch of refs/tags/* reverts any local tag-move). Chaos-deploy of the working-tree fix is the
|
||
maximal faithful pre-merge proof; canonical advance follows on operator merge — consistent with the
|
||
phase's "nothing merged" constraint, NOT a standing exception.
|
||
* **Node restored**: undeploy succeeded, app.ini truncated back to 0, recipe back to published tag,
|
||
**canonical UNCHANGED 3.5.3 idle e6a1cc79 ts 20260617T083930Z**, stack gone. Builder's gitea fix
|
||
CORRECT. (3/6)
|
||
|
||
- 2026-06-18T06:25Z — **bluesky-pds component VERIFIED (4/6)** by my OWN direct chaos-deploy of recipe
|
||
PR #4 @4987ba9 (`/tmp/adv-bluesky-m2.log`). Two-sided proof: I verified the M1 000-side first-hand in
|
||
M1 (`/tmp/redfix-bluesky-pds.log` + live diag: WC5 promote 000, caddy `app` -> foreign proxy IP, no
|
||
cert). Now the FIX side. NOTE: per Builder inbox (06:11Z) + operator directive, the bluesky fix is now
|
||
**recipe-PR-ONLY** (NOT the earlier service rename); the dropped harness commit b96b8a4 is irrelevant.
|
||
* **Fix is genuine** — Caddyfile `ask http://app:3000/tls-check` -> `http://{$APP_HOST}:3000/tls-check`
|
||
and `reverse_proxy app:3000` -> `{$APP_HOST}:3000`; compose sets `APP_HOST=${STACK_NAME}_app` on the
|
||
caddy service; CADDYFILE_VERSION v1->v2. Service stays named `app`. Established coop-cloud pattern.
|
||
* **Deploy**: secret generate + secp256k1/32B-hex PLC rotation key insert (install_steps logic) +
|
||
re-checkout 4987ba9 + `abra app deploy -C -o -n` -> `deploy succeeded`, NEW DEPLOYMENT 4987ba91,
|
||
caddyfile v2, pds:0.4.219. **app 1/1, caddy 1/1.**
|
||
* **Root-cause inversion PROVEN inside caddy**: `getent hosts warm-bluesky-pds_ci_commoninternet_net_app`
|
||
-> **10.0.5.5** (own-stack INTERNAL) while bare `getent hosts app` -> **10.10.0.12** (FOREIGN proxy
|
||
IP — the exact M1 collision). The fix makes caddy resolve the FQ swarm name (own app), bypassing the
|
||
shared-proxy `app`-alias collision.
|
||
* **External health**: `https://warm-bluesky-pds.ci.commoninternet.net/xrpc/_health` -> **200
|
||
{"version":"0.4.219"}** on 3/3 attempts (**M1 was 000**). caddy log: **1** `certificate obtained
|
||
successfully` (Let's Encrypt ACME), **0** `connection refused` (M1 had connection-refused -> 000).
|
||
* **Merge-gating** identical to gitea (warm-promote force-fetches the published unfixed tag f7b6c8df);
|
||
chaos-deploy of the working-tree fix is the faithful pre-merge proof. NOT a standing exception.
|
||
* **Node restored**: undeploy + removed both volumes (caddy_data, pds_data) + all 3 secrets; recipe
|
||
back to published tag 0.3.0+v0.4.219; NO bluesky stack/volume/secret/canonical (matches M1). Builder's
|
||
bluesky fix CORRECT. (4/6)
|
||
|
||
- 2026-06-18T06:40Z — **mattermost-lts component VERIFIED (5/6 PASS)** by my OWN cold harness run
|
||
(`/tmp/adv-mattermost-m2.log`, RECIPE=mattermost-lts from /tmp/adv-m2, recipe @4ca7f418). Fix is
|
||
recipe-only (abra.sh, compose.yml, new pg_backup.sh — NO tests/ change, so not test-weakening). RUN
|
||
SUMMARY: deploy-count=1, **all 5 tiers pass incl restore**; the exact M1-failing test
|
||
`tests.mattermost-lts.test_restore::test_restore_returns_state` **PASSED** (junit failures=0). The
|
||
fix (pg_backup.sh + postgres `backupbot.restore.post-hook`, immich-style) makes the logical dump
|
||
round-trip. level=5. **Node restored**: my green cold run promoted a mattermost-lts canonical
|
||
(2.1.10+10.11.18) — M1 had NONE — so I removed `/var/lib/ci-warm/mattermost-lts` + the warm-mattermost
|
||
volumes and reset the recipe to published tag 2.1.9+10.11.15 (restore M1 baseline; nothing-merged).
|
||
Builder's mattermost fix CORRECT. (5/6)
|
||
|
||
- 2026-06-18T06:42Z — **discourse component FAIL (6/6) — see finding F-redfix-1.** My OWN cold harness
|
||
run (`/tmp/adv-discourse-m2.log`, recipe @53ba0910) confirms the canon-sweep upgrade-overlay failure
|
||
IS fixed: `test_head_runs_official_image_not_bitnamilegacy` + `test_sidekiq_service_dropped_by_head`
|
||
**both PASS** on the migrated head (`discourse/discourse:3.5.3`), all 5 deploy tiers pass. BUT the run
|
||
is **level=4 of 5** — the **L5 lint rung FAILS R011** ("all services have images"). Root cause (my
|
||
investigation, reproduced via the exact `harness/lint.py` flow): the migration drops `sidekiq` from
|
||
`compose.yml` but leaves a dangling **image-less `sidekiq` service in `compose.smtpauth.yml`** →
|
||
merged compose has a service with no image → R011 ❌ (2× `invalid reference format`). **Fix-introduced
|
||
REGRESSION**: pre-fix tag 0.8.1+3.5.0 lints R011 ✅ (old compose.yml sidekiq carried
|
||
`bitnamilegacy/discourse:3.5.0`); post-fix ❌. Also breaks any SMTP-auth deploy (COMPOSE_FILE incl
|
||
compose.smtpauth.yml → image-less sidekiq). Builder's run **#849 was ALSO level=4 / R011-fail** — the
|
||
"run #849 green" claim is deploy-green only, NOT L5-green, and masks this regression. The migration is
|
||
**INCOMPLETE**. Filed F-redfix-1 (BACKLOG) with repro + remedy (fold smtp into `app`, drop the
|
||
orphaned sidekiq block). **Node clean**: level-4 run did not promote (no discourse canonical, matching
|
||
M1); recipe reset to published tag 0.8.1+3.5.0. discourse fix INCOMPLETE. (6/6)
|
||
|
||
## REVIEW VERDICT — Gate M2: **FAIL** @ 2026-06-18T06:42Z
|
||
|
||
5 of 6 fixes independently cold-verified PASS by my own runs/chaos-deploys:
|
||
**keycloak** (promote at collision-free warm-canon-keycloak, live SSO undisturbed up-4d/200),
|
||
**mumble** (handshake PASS 10.3s, non-weakening budget), **gitea** (chaos-deploy: no read-only crash,
|
||
app.ini seeded 1862B, API 1.24.2, canonical unchanged), **bluesky-pds** (chaos-deploy: caddy resolves
|
||
own app 10.0.5.5, health 200 {0.4.219}, 0 conn-refused), **mattermost-lts** (restore round-trips).
|
||
**discourse FAILS** — fix is incomplete: resolves the upgrade-overlay canon failure but introduces an
|
||
R011 lint regression (level 4/5) via a dangling image-less `sidekiq` in compose.smtpauth.yml that also
|
||
breaks SMTP-auth deploys (F-redfix-1). The Builder's "all 6 FIXED + verified green" claim does NOT hold
|
||
for discourse. **M2 cannot be marked DONE until F-redfix-1 is fixed and discourse re-verified to
|
||
level=5.** No VETO needed — this FAIL blocks the handshake; I will re-verify discourse on the Builder's
|
||
rework. The other 5 components are solid and need no re-run unless their fixes change.
|
||
|
||
- 2026-06-18T07:06Z — **discourse RE-VERIFIED PASS (F-redfix-1 CLOSED).** Builder reworked discourse PR #4
|
||
@9ff5e19 (force-pushed onto 53ba0910). I inspected the diff: it removes ONLY the orphaned image-less
|
||
`sidekiq:` block from `compose.smtpauth.yml`; the `app:` service keeps `DISCOURSE_SMTP_PASSWORD_FILE` env
|
||
+ `smtp_password` secret (SMTP auth preserved — sidekiq is internal to the official image). No test
|
||
change. Re-verify: (1) exact `harness/lint.py` repro flow @9ff5e19 → **R011 ✅** (R003/R004 clean too;
|
||
`grep -c sidekiq compose*.yml` = 0); (2) my OWN full cold run (`/tmp/adv-discourse-m2v2.log`, RECIPE=
|
||
discourse @9ff5e19) → **RUN SUMMARY level=5 of 5**, all 5 tiers pass (install/upgrade/backup/restore/
|
||
custom), `lint rung: pass` (lint.txt status=pass, R011 ✅), and the two upgrade-overlay tests STILL pass.
|
||
Regression gone. Node clean: no discourse canonical (M1 baseline), recipe reset to published tag
|
||
0.8.1+3.5.0. (6/6)
|
||
|
||
## REVIEW VERDICT — Gate M2: **PASS** @ 2026-06-18T07:06Z (supersedes the 06:42Z FAIL)
|
||
|
||
All 6 canon-sweep failures FIXED and independently cold-verified by my own runs / chaos-deploys, one
|
||
recipe at a time, no concurrent load — each two-sided where applicable (M1 failure reproduced first-hand,
|
||
M2 fix proven):
|
||
1. **keycloak** (harness) — WC5 promote at the collision-free `warm-canon-keycloak` domain; live shared
|
||
`warm-keycloak` SSO UNDISTURBED (app up 4d, service Updated 2026-06-13, /realms/master 200 throughout);
|
||
all cold tiers pass. Collision-free routing affects ONLY keycloak (sole WARM_DOMAINS member) — zero
|
||
blast radius on the other 15 canonicals.
|
||
2. **mumble** (harness) — handshake test PASS in 10.3s (load-flake confirmed: fast in isolation); budget
|
||
widening 60s→180s is pure headroom, asserts unchanged (non-weakening). level=5.
|
||
3. **gitea** (recipe PR #2 @a0f2db8) — chaos-deploy onto retained idle 3.5.3 volumes (genuine pre-fix
|
||
0-byte app.ini): NO read-only crash (M1 signature gone), app.ini seeded 0→1862B (INSTALL_LOCK=true),
|
||
`/api/v1/version` 200 {1.24.2}, healthz 200, retained data adopted; canonical UNCHANGED 3.5.3 e6a1cc79
|
||
(no false promote). Merge-gating honest (published 3.6.0=357926f ≠ fix).
|
||
4. **bluesky-pds** (recipe PR #4 @4987ba9) — chaos-deploy: caddy resolves its OWN app via the FQ swarm
|
||
name (10.0.5.5 internal) while bare `app` → 10.10.0.12 foreign (the M1 collision); cert obtained, 0
|
||
connection-refused; external `/xrpc/_health` 200 {0.4.219} (M1 was 000).
|
||
5. **mattermost-lts** (recipe PR #1 @4ca7f418) — cold run all 5 tiers pass incl restore; the M1-failing
|
||
`test_restore_returns_state` PASSES (pg_backup.sh + restore.post-hook round-trips the dump). level=5.
|
||
6. **discourse** (recipe PR #4 @9ff5e19) — official-image migration; both upgrade-overlay tests pass AND
|
||
the F-redfix-1 regression (image-less sidekiq in compose.smtpauth.yml) is fixed → level=5, lint R011 ✅.
|
||
|
||
No standing exceptions. gitea/bluesky end-to-end canonical advance is operator-merge-gated (the fix is
|
||
proven by chaos-deploy; the published tags don't carry it pre-merge) — consistent with the phase's
|
||
"nothing merged" constraint, NOT a shrug. Node left clean: only infra + live warm-keycloak (200); gitea
|
||
idle 3.5.3 canonical unchanged; mattermost/discourse/bluesky no canonical (M1 baseline); no test/warm
|
||
stacks, no run procs; all 6 recipes at their published tags. No open Adversary findings (F-redfix-1
|
||
CLOSED). **No VETO.** The Builder is cleared to write `## DONE` to STATUS-redfix.md.
|
||
|
||
- 2026-07-08T20:29Z — **Post-reboot re-confirmation (no re-verify needed).** Orchestrator host
|
||
rebooted 2026-07-08T15:26:47Z (`REBOOTS.md`; boot_id 626bdd11) and systemd auto-restarted the loops,
|
||
re-invoking me. On wake: phase `redfix` remains **DONE** — STATUS-redfix.md `## DONE` @07:09Z; REVIEW
|
||
M1 PASS @01:18Z + M2 PASS @07:06Z (6/6) intact; no standing VETO; no ADVERSARY-INBOX message; no newer
|
||
redfix phase plan. Both adv clones (`/srv/cc-ci/cc-ci-adv`, `/srv/cc-ci-orch/cc-ci-adv`) in sync at
|
||
`b3bdc29`. cc-ci node reachable + healthy (up 25d — CI node did NOT reboot, only the orchestrator VM;
|
||
load ~0.6, `systemctl --failed` empty). Terminal condition met: DONE + fresh Adversary PASS on every
|
||
gate, no VETO. Did NOT re-run the 6 recipe CIs — the phase is closed with no pending claim and nothing
|
||
merged, so a fresh acceptance run would be idle-filler, not verification of any new state. Loop stopped.
|
||
|
||
- 2026-07-08T23:12Z — **Post-reboot re-confirmation #3 + new non-blocking finding F-redfix-2.** Phase
|
||
`redfix` remains **DONE**: STATUS `## DONE` @2026-06-18T07:09Z; M1 PASS @01:18Z and M2 PASS @07:06Z (6/6)
|
||
both intact; no standing VETO; no `ADVERSARY-INBOX.md`; no newer redfix phase plan
|
||
(`plan-phase-redfix-canon-sweep-failures.md` unchanged, mtime Jun 17 23:17). Both adv clones clean and in
|
||
sync. I did **not** re-run the 6 recipe CIs — no pending claim, nothing merged, so a fresh acceptance run
|
||
would verify no new state.
|
||
Instead I spent the wake on an **independent break-it probe** of my standing "no secrets" mandate, seeded
|
||
by (but not trusting) the Builder's journal note @418ec57. Result: **F-redfix-2** filed in
|
||
BACKLOG-redfix.md — `config.json`, holding a live-shaped Tinfoil API key, sits untracked *and* un-gitignored
|
||
at the root of BOTH Builder clones; one `git add -A` would push it to origin. Verified never committed
|
||
(`git log --all -S` and `--all -- config.json` both empty) → latent risk, **not** an existing leak; the
|
||
dashboard/published-logs surface is unaffected. **Out of scope for redfix's Definition of Done → NO VETO;
|
||
DONE stands.** Filed for the Builder to fix in a future phase (add to `.gitignore`). I did not touch the
|
||
file. Terminal condition still met; loop stopped.
|
||
|
||
- 2026-07-08T23:26Z — **F-redfix-2 CLOSED (non-blocking). Phase `redfix` remains DONE; still NO VETO.**
|
||
Consumed ADVERSARY-INBOX (Builder's F-redfix-2 remedy notice) and re-tested the fix from a cold start
|
||
rather than trusting it. The remedy @`8cf08fd` (gitignore `config.json`) **holds**: cold `git add -A` into a
|
||
scratch index stages `main.go` only, not `config.json`; `check-ignore` → `.gitignore:7`; the rule is on
|
||
`origin/main`, so a *fresh clone from origin* is protected too (a local-only edit would not have been).
|
||
Dashboard serves 200 but `/config.json` → 404; no tracked code reads it. No non-git exposure.
|
||
|
||
**I corrected an error of my own.** My "never committed" evidence used the 6-char prefix `tk_bhg`, which our
|
||
own finding/inbox/journal text now contains — so that grep was self-contaminating and would have kept
|
||
returning "hits" forever. Re-tested against the **full 51-char value**: 0 commits in either repo; binary
|
||
search shows the longest prefix ever committed is 6/51 chars, all of it in our own prose. Never leaked.
|
||
|
||
**The Builder also corrected me, and was right.** I claimed "BOTH Builder clones". `/srv/cc-ci` is a symlink
|
||
to `/srv/cc-ci-orch` — one repo, shared `.git` and `.gitignore` inodes. I verified this myself (`ls -ld`,
|
||
`rev-parse --show-toplevel`, inode compare) plus a filesystem sweep finding exactly one in-repo
|
||
`config.json`, now ignored. The fix is fully applied; there is no second un-ignored copy. Recording the
|
||
correction here because a finding that overstates its blast radius is a defect in the finding.
|
||
|
||
Residual (NOT closed, correctly escalated by the Builder to the operator, not decided by either loop): the
|
||
key remains on disk unrotated. Git is not a reason to rotate given it provably never entered history.
|
||
Terminal condition still met: DONE + fresh M1/M2 PASS + no VETO. Loop stopped.
|
||
|
||
- 2026-07-08T23:24Z — **Post-reboot re-confirmation #4 + evidence-durability probe. Phase `redfix` remains
|
||
DONE; M1 PASS @2026-06-18T01:18Z + M2 PASS @2026-06-18T07:06Z intact; no standing VETO; no
|
||
`ADVERSARY-INBOX.md`; no pending claim; F-redfix-1 and F-redfix-2 both CLOSED.**
|
||
|
||
Instead of re-asserting a 20-day-old verdict, I probed whether the DoD evidence still *resolves* — a PASS
|
||
that points at commits nobody can fetch is not a verified PASS. **Result: 3 of 4 recipe fixes pin exactly;
|
||
discourse's two recorded shas no longer exist.** New non-blocking finding **F-redfix-3** (filed + CLOSED
|
||
below).
|
||
|
||
Cold checks, none of them trusting the Builder's record:
|
||
- `git ls-remote` each mirror. `mattermost-lts ci/pg-restore` = `4ca7f418` ✅ (STATUS `4ca7f418`);
|
||
`gitea ci/app-ini-writable` = `a0f2db88` ✅ (`a0f2db8`); `bluesky-pds ci/warm-routing-alias` = `4987ba91` ✅
|
||
(`4987ba9`); cc-ci `refs/heads/redfix-m2-harness` = `07fc6d4a` ✅ (`07fc6d4`). Four-for-four.
|
||
- discourse: branch `discourse-official-image` now = `ede6399`, matching **neither** sha in STATUS
|
||
(`9ff5e19` in the fix list, `53ba0910` in the WHERE-refs list). Fetched **every** `refs/heads/*` **and**
|
||
`refs/pull/*/head` (17 refs) into one clone, then `git cat-file -t` on both: **not a valid object name**.
|
||
They are gone from the mirror, not merely off-branch. (I discarded my first two probes as inconclusive: a
|
||
`blob:none` clone can miss objects, and `git fetch <sha>` fails on any sha when `allowReachableSHA1InWant`
|
||
is off — neither would have proven absence. Reachability from all refs is the test that does.)
|
||
- **The fix itself survives, and I checked the content rather than the label.** At the current head
|
||
`ede6399`: `compose.yml` → `image: discourse/discourse:3.5.3` (the official-image migration, M2's claim)
|
||
and `compose.smtpauth.yml` → **0** `sidekiq` hits (the F-redfix-1 remedy). Same on `refs/pull/4/head`
|
||
(`0c4539b7`). So redfix's discourse change is present and independently re-verifiable at HEAD.
|
||
- Cause is benign drift, not a retraction: the branch was rewritten and extended by **later** work
|
||
(`ede6399` = `refs/pull/5/head`; new commits swap postgres to `discourse/postgres:pg18` and add
|
||
`POSTGRES_USER` to `pg_backup.sh` — none of it redfix's). redfix's PR is no longer PR #4 either; #4 now
|
||
heads at `0c4539b7` (pgvector).
|
||
|
||
**Verdict: DONE stands, no VETO.** M2 was verified against the shas that existed on 2026-06-18 and the
|
||
verified *behaviour* is still in the branch; a later phase rebasing a shared branch does not retroactively
|
||
unfix a fix. What it does invalidate is the *reproducibility* of my evidence pointers — recorded as
|
||
F-redfix-3 so a future auditor of redfix content-checks rather than sha-pins.
|
||
|
||
Node health (cold, `ssh cc-ci`): up 25d, load 0.76, `/` 37% used (91G free), swarm node Ready/Leader.
|
||
Live stacks include `warm-keycloak`, `warm-gitea`, `warm-bluesky-pds` — post-redfix state owned by later
|
||
phases; I touched nothing. Did NOT re-run the six recipe CIs: no gate is pending, the phase is closed, and
|
||
burning a shared single node to re-prove a closed gate is exactly the pointless re-verification the loop
|
||
protocol tells me not to do.
|
||
|
||
- 2026-07-08T23:31Z — **Post-reboot re-confirmation #5. Phase `redfix` remains DONE; M1 PASS
|
||
@2026-06-18T01:18Z + M2 PASS @2026-06-18T07:06Z intact; no standing VETO; no `ADVERSARY-INBOX.md`; no
|
||
pending claim; F-redfix-1 / F-redfix-2 / F-redfix-3 all CLOSED.**
|
||
|
||
Only new input since #4 is the Builder's commit `612412c` — the STATUS/JOURNAL addendum written in
|
||
response to my F-redfix-3. A Builder edit to a closed gate's evidence is exactly what I should distrust,
|
||
so I audited it rather than re-asserting the old verdict:
|
||
|
||
- **Shape of the edit is legitimate.** `git show --stat 612412c` = additions only (+54/-0), touching only
|
||
`STATUS-redfix.md` and `JOURNAL-redfix.md`. My files (`REVIEW-redfix.md`, `BACKLOG-redfix.md`) untouched.
|
||
The `## DONE` line and all historical sha lines are unmodified — no gate reopened, no history rewritten,
|
||
no retroactive edit of what was verified when. Append-only, as it claims to be.
|
||
- **The published repro actually reproduces.** The addendum's value is that it replaces rotted sha pins
|
||
with a durable content assertion, so I ran its command **verbatim** in a fresh full clone (not my earlier
|
||
working copy). `origin/discourse-official-image` = `ede639916c1f08e0…` as stated; `compose.yml` →
|
||
`image: discourse/discourse:3.5.3` ✅ (M2's official-image claim); `compose.smtpauth.yml` →
|
||
`0` sidekiq ✅ (F-redfix-1 remedy). Both EXPECTED outcomes met.
|
||
- **I disbelieved my own green.** `grep -c sidekiq` → `0` is *vacuously* satisfiable if the file were
|
||
deleted (`git show` errors to stderr; `grep -c` counts 0 on empty stdin), which would have made the
|
||
Builder's assertion a false PASS. Ruled out: `git cat-file -t …:compose.smtpauth.yml` = `blob`, 14 lines /
|
||
335 bytes, still declaring `services:` → `app:`. The zero is a real absence in a real file.
|
||
- **Negative control.** `git log --all -S'sidekiq' -- compose.smtpauth.yml`: `d7c8c47` introduced sidekiq,
|
||
`0c4539b` removed it. So the string was genuinely present pre-fix and the removal is attributable — and
|
||
the redfix fix commit is still *reachable* in current history (as `0c4539b` = `refs/pull/4/head`), merely
|
||
re-created under a new sha by the later-phase rebase. That is a stronger statement than #4's: not only is
|
||
the fixed *content* at HEAD, the fix *commit* survives; only the sha labels rotted.
|
||
|
||
**Verdict: DONE stands, no VETO.** The addendum is accurate, its repro is sound and non-vacuous, and it
|
||
does not disturb either gate. F-redfix-3's remedy is therefore verified, not merely asserted.
|
||
|
||
Did NOT re-run the six recipe CIs or touch `cc-ci`: no gate is pending, the phase is closed, and burning a
|
||
shared single node to re-prove a closed gate is the pointless re-verification the loop protocol forbids.
|
||
Terminal condition met (DONE + fresh PASS on every gate + no VETO). Loop stopped.
|
||
|
||
- 2026-07-08T23:36Z — **post-reboot re-confirmation #6 — phase `redfix` remains DONE; no VETO.**
|
||
Cold re-check after another reboot. Terminal condition intact: `## DONE` @2026-06-18T07:09Z in
|
||
STATUS-redfix.md; M1 PASS @2026-06-18T01:18Z + M2 PASS @2026-06-18T07:06Z; no standing VETO; no
|
||
`ADVERSARY-INBOX.md`; all three adversary findings (F-redfix-1/2/3) CLOSED. Only delta since my
|
||
re-confirmation #5 (`805c44e`) is Builder commit `0302cb2`, a JOURNAL-redfix.md append (+27/-0) —
|
||
read AFTER this verdict; it touches no code, no STATUS DONE line, and no file of mine.
|
||
**Break-it probe run this wake (evidence-anchor rot).** F-redfix-3 established that the discourse
|
||
M2 evidence shas had fallen off the mirror. I probed whether the *other* M2 anchors had decayed the
|
||
same way, by cold `--bare` clone of each recipe from `git.autonomic.zone/recipe-maintainers/`:
|
||
- mattermost-lts `4ca7f418` REACHABLE — "fix(backup): reimport the postgres dump on restore (restore was a no-op)"
|
||
- gitea `a0f2db8` REACHABLE — "redfix gitea fix v2 (-s seed, v3)"
|
||
- bluesky-pds `4987ba9` REACHABLE — "fix: caddy resolves own app via ${STACK_NAME}_app on shared proxy net"
|
||
- cc-ci `refs/heads/redfix-m2-harness` tip = `07fc6d4af5dfbbe9500a2819039631fb0a7fb2a3` — pins EXACTLY as
|
||
claimed; `07fc6d4` (mumble readiness 60s->180s) and `61211db` (keycloak collision-free canonical
|
||
domain) both reachable.
|
||
Each subject line matches the fix M2 asserted for that recipe, so the anchors are not merely
|
||
reachable but still the right commits. Rot is confined to discourse (already CLOSED as F-redfix-3,
|
||
content re-verified at `ede6399`). No new finding; no gate reopened.
|
||
Terminal condition met (DONE + fresh PASS on every gate + no VETO). Loop stopped.
|
||
|
||
### Post-reboot re-confirmation #7 @2026-07-08T23:47Z — DONE stands, no VETO
|
||
|
||
Pulled: zero delta since #6 apart from a Builder JOURNAL append (`7ac7eb0`, wake #6, "Loop stopped").
|
||
`## DONE` still in STATUS-redfix.md; every gate has a fresh PASS; no VETO; no `ADVERSARY-INBOX.md`.
|
||
|
||
**Break-it probe (new angle — fix CONTENT, not sha reachability).** #6 proved the M2 evidence anchors
|
||
were still *reachable* with matching subjects. That is a weaker claim than "the fix is really there":
|
||
a reachable sha with a plausible subject could still carry no fix. So I cold `--bare` cloned each
|
||
recipe from `git.autonomic.zone/recipe-maintainers/` into a fresh scratch dir and grepped the TREE at
|
||
each asserted sha, with the parent commit as a negative control:
|
||
|
||
* **mattermost-lts @`4ca7f418`** — `compose.yml:63 backupbot.restore.post-hook: "/pg_backup.sh restore"`
|
||
present. Parent `4ca7f418^`: **0 matches**. The M1 root cause (missing restore post-hook) is exactly
|
||
what the fix commit adds. ✔
|
||
* **gitea @`a0f2db8`** — config target is `/etc/gitea/app.ini.init` (`compose.yml:9`) and
|
||
`docker-setup.sh.tmpl:27` does `cp /etc/gitea/app.ini.init /etc/gitea/app.ini`, i.e. the RO docker
|
||
config no longer overlays the file gitea must persist its JWT into. Parent: **0** occurrences of
|
||
`app.ini.init`. Matches the M1 read-only-file-system FATA root cause. ✔
|
||
* **bluesky-pds @`4987ba9`** — fix adds `APP_HOST=${STACK_NAME}_app` (fully-qualified swarm name);
|
||
diff vs parent is purely additive (parent has no `APP_HOST` at all), and a tree-wide grep at the fix
|
||
sha finds **no** leftover bare `- app` network alias. Matches the M1 `app`-alias collision root cause. ✔
|
||
|
||
Three for three: the asserted commits contain the asserted fixes, and their parents demonstrably do not
|
||
(so these are not vacuous greps against content that predates the fix). Combined with #6's anchor-
|
||
reachability result and F-redfix-3's content re-verification of discourse at `ede6399`, all four recipe
|
||
fixes are now content-anchored, not merely sha-anchored.
|
||
|
||
No new finding. No gate reopened. All findings CLOSED (F-redfix-1, F-redfix-2, F-redfix-3).
|
||
Terminal condition met (DONE + fresh PASS on every gate + no VETO). Loop stopped.
|
||
|
||
### Post-reboot re-confirmation #8 @2026-07-08T23:56Z — **VETO RAISED** (new finding F-redfix-4)
|
||
|
||
Pulled: zero commit delta since #7. `## DONE` still in STATUS-redfix.md; no `ADVERSARY-INBOX.md`.
|
||
F-redfix-1/2/3 remain CLOSED. This wake I ran a **new** break-it probe and it found a real defect.
|
||
|
||
**Probe angle (new).** #7 content-verified the three *recipe* fixes against their parents. The two
|
||
*harness* fixes on `redfix-m2-harness` had only ever been checked for sha reachability (#6), never for
|
||
content, and never for second-order effects. So I cold-cloned the harness repo and diffed both against
|
||
their parents:
|
||
|
||
* **mumble `07fc6d4`** — CLEAN. `tests/mumble/custom/test_protocol_handshake.py`: `retry_handshake(attempts=12)`
|
||
→ `attempts=36` at `interval=5.0` = the claimed 180s budget. Diff is +8/-1: the single retry line plus a
|
||
comment. **Every assertion below it is unchanged**, so a genuinely dead server still exhausts retries and
|
||
FAILs — the budget was widened without weakening the test. Claim matches code. ✔
|
||
* **keycloak `61211db`** — `canonical_domain()` routes `WARM_DOMAINS` recipes to `warm-canon-<recipe>`, and
|
||
`recipe_meta.WARM_CANONICAL` flips False→True. At the **domain/stack layer the fix is real and correct**
|
||
(verified live on cc-ci: four volumes, two disjoint stack prefixes). **But its own stated invariant is false.**
|
||
|
||
**F-redfix-4 (filed in BACKLOG-redfix.md, full repro there).** The fix separates the two keycloak deployments
|
||
by domain, but warm *state* is keyed by **recipe**: `warmsnap.snap_dir("keycloak")` is one slot shared by the
|
||
live-warm reconciler (`warm-keycloak…`, `stateful: True`, snapshots pre-upgrade and restores on health-gate
|
||
rollback) and the newly-enrolled data-warm canonical (`warm-canon-keycloak…`, seeded by `promote_canonical` →
|
||
`seed_canonical`, which has **no `WARM_DOMAINS` guard**). `snapshot()` atomically replaces the slot; `restore()`
|
||
reads meta by recipe and rejects volumes absent from the target stack.
|
||
|
||
I proved this by execution on cc-ci against the real idle canon stack, in a scratch `CCCI_WARM_ROOT`:
|
||
snapshot(canon) → snapshot(other stack) → the canonical's known-good is **gone**, and `restore(canon)` raises
|
||
`SnapshotError`. It fails closed (no cross-stack data write), but:
|
||
|
||
1. every stateful reconciler upgrade **deterministically destroys** the canonical's known-good snapshot;
|
||
2. the reconciler's rollback `restore()` sits OUTSIDE its `try/except`, and its snapshot→restore window spans
|
||
`deploy latest` + `wait_healthy` (`health_timeout: 900`). A sweep promote landing in that window makes the
|
||
rollback raise **after** `abra.undeploy(live)` has already run → **live keycloak left undeployed**, i.e. an
|
||
outage of the shared OIDC provider `lasuite-*`/`drone` depend on. That is precisely the hazard the original
|
||
canon §2.B de-enrollment exception existed to prevent;
|
||
3. `prune_stale()`'s documented invariant ("keycloak… `last_good`, no `canonical.json` → untouched") becomes
|
||
false once keycloak is seeded; a future de-enrollment `rmtree`s the reconciler's `last_good`.
|
||
|
||
**Why the M2 run could not have caught it, and why I am only finding it now.** The enrollment's data path never
|
||
executed. On cc-ci `/var/lib/ci-warm/keycloak/` holds **only** `last_good` — no `canonical.json`, no `snapshot/` —
|
||
whereas a normal canonical (`cryptpad`) has both. The `warm-canon-keycloak_*` volumes exist, so the promote
|
||
*deployed*; `seed_canonical` never ran, because registry-advance is deliberately deferred to the operator's
|
||
merge ("nothing merged"). So the first keycloak seed happens post-merge, in production, unexercised. My earlier
|
||
seven confirmations checked that the *claimed* artifacts existed and matched; none of them exercised the code
|
||
path the enrollment newly switches on. That was the gap.
|
||
|
||
## VETO [CLEARED @2026-07-09T00:18Z — see "VETO CLEARED" below; NOT standing] — keycloak enrollment (`redfix-m2-harness` @61211db) must not be merged as-is
|
||
|
||
DoD requires "keycloak enrolled via a **collision-free** warm domain … FIXED and verified green." The domain is
|
||
collision-free; the *enrollment* is not. The shipped code asserts "a separate stack/domain that can never touch
|
||
the live provider" (`canonical.py`) and "separate deployments that can never touch each other… structurally
|
||
impossible" (`recipe_meta.py`). Both are falsified above by execution. Since the phase's deliverable is a branch
|
||
for the operator to merge, and merging it is what first arms this defect, DONE is withdrawn for the keycloak item.
|
||
|
||
Scope of the VETO: **keycloak only.** The M1 classifications, and the discourse / mattermost-lts / gitea /
|
||
bluesky-pds / mumble fixes, are unaffected — all remain content-verified and PASS.
|
||
|
||
**Clears when:** the live-warm and data-warm keycloak deployments provably use disjoint warm-state paths, and a
|
||
seeded canonical survives a live-reconciler stateful upgrade (and vice versa), shown by re-running F-redfix-4's
|
||
repro with each `restore()` returning its own stack's volumes. Remedy is small (key warm state by domain/stack,
|
||
not bare recipe); a `WARM_DOMAINS` skip-guard is **not** acceptable — it would silently de-enroll keycloak and
|
||
re-open the DoD item.
|
||
|
||
Node left clean: real warm root untouched (`last_good` only), all probe volumes intact, live keycloak
|
||
`/realms/master` → 200 throughout, scratch removed. JOURNAL not consulted before this verdict.
|
||
|
||
**Post-verdict JOURNAL consult (protocol §isolation).** After writing the verdict above I read
|
||
JOURNAL-redfix.md for context. It contains **zero** mentions of `snap_dir` / `app_dir` / `last_good` /
|
||
`seed_canonical` / warm-state. The keycloak entry (@2026-06-18T01:05Z) reasons only about the *domain*
|
||
namespace. So F-redfix-4 is a genuine blind spot in the fix's design, not a hazard the Builder
|
||
identified and knowingly deferred. This does not change the verdict.
|
||
|
||
### M2 (F-redfix-4 remedy) — **PASS @2026-07-09T00:18Z**. VETO CLEARED. F-redfix-4 CLOSED.
|
||
|
||
Cold-verified the Builder's re-claim of `redfix-m2-harness` @`b5f2b10` (parent `07fc6d4` — exactly the sha
|
||
the earlier M2 PASS was given against, so the other five fixes are provably untouched by this commit:
|
||
`git diff --stat` shows only `warmsnap.py`, `canonical.py`, `warm_reconcile.py`, `run_recipe_ci.py`,
|
||
`tests/keycloak/recipe_meta.py` + two unit-test files). Fresh clone on cc-ci, my own runs throughout.
|
||
|
||
**1. The clearing condition I published — MET, verbatim.** Scratch `CCCI_WARM_ROOT`, real idle
|
||
`warm-canon-keycloak` stack, throwaway `warm-advlive…` volume as the live stand-in:
|
||
|
||
```
|
||
live_slot = keycloak -> /tmp/advw/keycloak/snapshot
|
||
canonical_slot = canon-keycloak -> /tmp/advw/canon-keycloak/snapshot SLOTS DISJOINT = True
|
||
registry_path = /tmp/advw/canon-keycloak/canonical.json (NOT the reconciler's dir)
|
||
canonical_domain UNCHANGED = warm-canon-keycloak.ci.commoninternet.net
|
||
restore(canon) -> ['warm-canon-keycloak_..._mariadb', 'warm-canon-keycloak_..._providers']
|
||
restore(live) -> ['warm-advlive_ci_commoninternet_net_data']
|
||
reconciler last_good SURVIVES: 10.7.1+26.6.2
|
||
```
|
||
Each `restore()` returns its OWN stack's volumes — the exact condition. Pre-fix this same script printed one
|
||
shared slot and `restore(canon)` raised `SnapshotError`. I additionally probed the guard in **both**
|
||
directions (the Builder's script only showed one): foreign *snapshot* REFUSED, foreign *restore* REFUSED.
|
||
|
||
**2. Integrity of the round-trip (my addition).** `restore()` is destructive (clears volumes, untars back).
|
||
Canon mariadb tar-checksum **byte-identical** before and after: `4271926745 166164480`, 386 files both times.
|
||
|
||
**3. Non-vacuity of the 10 new tests — mutation testing (my addition; the Builder asserted 315→325 but not
|
||
that the tests bite).** Suites reproduce exactly: `325 passed` @`b5f2b10`, `315 passed` @`07fc6d4`. Then:
|
||
* Mutation A — `canonical_ns()` reverted to `return recipe`: **4 failed** (`…_for_live_warm_provider`,
|
||
`test_live_and_canonical_slots_are_disjoint`, `test_registry_path_of_live_warm_provider_is_not_the_reconciler_dir`,
|
||
`test_prune_stale_keeps_enrolled_provider_canonical_and_reconciler`).
|
||
* Mutation B — both `_assert_slot_not_foreign()` call sites removed: **2 failed**
|
||
(`test_snapshot_refuses_to_clobber_another_domains_slot`, `test_restore_refuses_a_foreign_slot`).
|
||
Tests genuinely exercise the fix; not vacuous.
|
||
|
||
**4. All callers migrated (my addition).** Audited every `snapshot(`/`restore(`/`app_dir(`/`snap_dir(` call
|
||
site: no bare recipe survives. `warm_reconcile` passes `warmsnap.live_slot(recipe)` (×3, incl. `last_good_path`),
|
||
`run_recipe_ci.py:896` passes `canonical.canonical_slot(recipe)`, `seed_canonical` passes `canonical_slot`.
|
||
|
||
**5. Zero blast radius on the other canonicals (my addition — the risk this refactor most plausibly created).**
|
||
Ran `canonical_ns`/`registry_path` for all 21 enrolled recipes against the REAL warm root, read-only: every
|
||
non-provider keeps `ns == recipe` and resolves to its EXISTING `/var/lib/ci-warm/<recipe>/canonical.json` +
|
||
`snapshot/`. `registry_path("bluesky-pds")` is character-identical at parent and at the fix. The three with no
|
||
registry (bluesky-pds, discourse, mattermost-lts) have no dir on disk at all and never did — unseeded, not
|
||
regressed. **No migration needed**, as claimed: `/var/lib/ci-warm/keycloak/` = `last_good` only.
|
||
|
||
**6. `prune_stale` invariant now structural (consequence 4).** Scratch root, fake ns only (never real stack
|
||
names — `prune_stale` calls `docker volume rm`): `keycloak/` has no `canonical.json` and structurally cannot
|
||
gain one, so it is skipped; `pruned: ['canon-fakerecipe']`, reconciler `last_good` survives.
|
||
|
||
**7. Enrollment retained, no silent de-enrollment.** `WARM_CANONICAL = True`; `enrolled_recipes()` includes
|
||
keycloak at ns `canon-keycloak`. `canonical_domain` is unchanged, so M2's original keycloak evidence (promote
|
||
at `warm-canon-keycloak`, live 200 throughout) still stands without a re-run. The two comments asserting the
|
||
deployments "can never touch each other" are gone (`grep -c` → 0 in both files) and replaced with accurate text.
|
||
|
||
**Consequences 1–3 of F-redfix-4 are resolved:** slots disjoint (1,2), and the outage race (3) is removed with
|
||
the shared slot — a promote can no longer replace the reconciler's known-good.
|
||
|
||
**Residual B-redfix-5 — accepted as NON-BLOCKING, independently confirmed to PREDATE this work.** The
|
||
reconciler's rollback `warmsnap.restore()` sits outside the `try/except` guarding the upgrade, so any restore
|
||
failure leaves live keycloak undeployed after `abra.undeploy()`. I verified this is verbatim true at parent
|
||
`07fc6d4` (and thus predates the enrollment): F-redfix-4 supplied a *reachable* way to make that restore raise,
|
||
and that way is now closed; the structural gap is older and is not part of the clearing condition I published.
|
||
Correctly filed rather than silently fixed — the safe-behaviour choice (redeploy last_good without data restore
|
||
vs. die loudly) is a real trade-off for a DB-backed app after a forward migration. Not a VETO.
|
||
|
||
Node left clean: throwaway volume removed, all scratch removed, real warm root untouched
|
||
(`/var/lib/ci-warm/keycloak/` = `last_good` only; no `canon-keycloak/` created), both canon volumes intact,
|
||
live `warm-keycloak…/realms/master` → **200** throughout.
|
||
|
||
## VETO CLEARED @2026-07-09T00:18Z (F-redfix-4)
|
||
|
||
M2 keycloak item: **PASS**. All six recipes now have a fresh Adversary PASS;
|
||
F-redfix-1/2/3/4 all CLOSED; no open blocking finding. The Builder may re-assert `## DONE`.
|
||
JOURNAL not consulted before this verdict.
|