recipe-maintainers/cc-ci
0
0
Fork 0
Code Issues Pull Requests 1 Actions Packages Projects Releases Wiki Activity
Files
1182aa0c2c90b1fb3fff12ab734aa586af725db0
cc-ci/machine-docs/REVIEW-redfix.md

65 KiB
Raw Blame History

REVIEW — phase redfix (Adversary)

Phase SSOT: /srv/cc-ci/cc-ci-plan/plan-phase-redfix-canon-sweep-failures.md Mission: investigate every canon-sweep failure (discourse, mattermost-lts, mumble, bluesky-pds, gitea, keycloak), isolate → root-cause → classify (flake vs genuine; recipe vs test vs warm-machinery vs load) → FIX each via a recipe PR or harness improvement → verify green. No standing exceptions. Nothing merged.

Gates:

  • M1 — all six investigated in isolation, classified with evidence. Adversary cold-verifies: claimed flake = reproducibly green in isolation (and red under load); claimed recipe defect = genuinely the recipe (not a stale test / harness artifact); claimed warm-machinery bug = in cc-ci.
  • M2 — all six FIXED + verified green (recipe PR via !testme; harness/cc-ci PR via the harness; flake-stabilization green under load). All six promote/pass. No standing exception. Nothing merged.

DONE = Builder writes ## DONE only after M1+M2 fresh Adversary PASS here.

Verdicts

M1 — investigate + isolate + classify: PASS @ 2026-06-18T01:18Z

Gate claim: claim(redfix-M1) commit 0a06c41 (@00:25Z). Verified from a COLD START on cc-ci with my OWN isolation re-runs (one recipe at a time, no concurrent load) — NOT the Builder's logs. Isolation discipline honored: verdict formed from the phase plan (SSOT), the recipe code / git history, the verification info in STATUS, and my own cold acceptance runs; I did NOT read JOURNAL-redfix.md before writing this verdict.

All six classifications are CORRECT. Evidence per recipe (full detail in the verification log below):

Recipe My independent reproduction Classification — verified
discourse my isolation run /tmp/adv-discourse.log: install/backup/restore/custom PASS, upgrade FAIL on the 2 PR-faithfulness overlay asserts; converged in minutes, no FATA/rc=142/wedge stale/PR-specific cc-ci OVERLAY test (canon "timeout" root-cause was WRONG — confirmed). Recipe deploys+serves fine. ✔
mattermost-lts my isolation run /tmp/adv-mattermost.log: restore FAIL deterministically (relation "ci_marker" does not exist, 91s, isolated) genuine RECIPE defect — no backupbot.restore.post-hook; NOT the canon "loaded-node race." ✔
mumble my isolation run /tmp/adv-mumble.log: ALL 5 tiers GREEN incl test_handshake_completes_with_channel_presence; promote OK load/timing FLAKE — green in isolation (a recipe defect would red deterministically; it didn't). ✔
bluesky-pds my isolation run /tmp/adv-bluesky.log + live caddy diag: cold GREEN, warm promote 000 deterministic; getent app→10.10.0.4 (foreign proxy), own app 10.0.5.6 never resolved; caddy log cycles dial 10.10.0.{4..12}:3000 refused genuine recipe ROUTING defect (bare app + caddy on shared proxy), NOT cc-ci promote-machinery (it correctly refused to promote), NOT flake. (Reverses the plan's "warm-machinery" prior — confirmed against it.) ✔
gitea my isolation run /tmp/adv-gitea.log + container crash log: cold GREEN, warm advance crash-loops 0/1; LoadCommonSettings() [F] … error saving JWT Secret … "/etc/gitea/app.ini": read-only file system; canonical correctly stayed 3.5.3 (promote timed out, refused) genuine RECIPE defect (3.6.0 JWT save vs read-only app.ini docker-config mount; /etc/gitea is a writable volume but the app.ini file is the RO config). ✔
keycloak code-verified: canonical.canonical_domain('keycloak')→warm.stable_domain→warm-keycloak.ci.commoninternet.net == warm.WARM_DOMAINS['keycloak'] (warm.py:47 documents the equality); live keycloak 200 on /realms/master HARNESS defect (data-warm canonical domain collides with the live-warm OIDC provider; no collision-free namespace). ✔

No defects in the classification work. No VETO. Node verified clean before AND after my runs (only infra

  • live warm-keycloak; gitea restored to undeployed idle 3.5.3, volumes retained, canonical commit e6a1cc79 unchanged; warm-keycloak healthy throughout). M1 PASS — Builder cleared to proceed to M2. (M2 will re-verify each FIX green; this PASS is for the investigation/classification gate only.)

(prior placeholder removed)

Adversary verification log

  • 2026-06-17T23:18Z — Phase redfix opened. Refreshed phase plan + plan.md §6.1. Cold access to cc-ci confirmed (ssh cc-ci: host nixos, uptime 4d, systemctl --failed empty, load ~0.8). No Builder state files (STATUS/BACKLOG/JOURNAL-redfix.md) yet; no gate claimed. Idling for the first claim.

  • 2026-06-18T00:10Z — Non-contending pre-staging (M1 NOT yet claimed; Builder mid-investigation: gitea isolation running, keycloak pending). Stayed OFF the swarm to avoid contaminating the Builder's isolation runs. Independently corroborated two deterministic static claims via pure code reads on cc-ci (no deploys):

    • mattermost-lts (recipe @ 2.1.9+10.11.15): postgres svc has backupbot.backup.pre-hook (pg_dump → /var/lib/postgresql/data/postgres-backup.sql), backup.post-hook (rm dump), backup.path=/var/lib/postgresql/data/ (hot live PGDATA) — and NO backupbot.restore.post-hook. immich (passes) uses dump-only backup.volumes.postgres.path: backup.sql + restore.post-hook: /pg_backup.sh restore. Corroborates "genuine recipe defect — no restore round-trip." ✔ pre-staged.
    • discourse (recipe @ 0.8.1+3.5.0 = bitnamilegacy/discourse:3.5.0 + sidekiq): overlay tests/discourse/test_upgrade.py is a phase-prevb PR-faithfulness test asserting app image == official discourse/discourse:3.5.3 AND sidekiq dropped — only true on an unreleased PR head, not the latest release the canon sweep deploys. So it red-by-construction in the sweep. Corroborates "stale/PR-specific overlay test, not flake/timeout/recipe-deploy." ✔ pre-staged.
    • STILL OWED before any M1 PASS: my OWN cold isolation run of discourse to confirm the re-classification from the original canon hypothesis ("cold-deploy timeout, ~51-min wedge") to "deploys+serves fine, only the overlay test reds." Will run when M1 is claimed and the swarm is free (Builder not deploying). Same for bluesky app-alias collision (needs live caddy/getent diag). These are NOT verdicts — formal M1 PASS/FAIL awaits the Builder's gate claim.

  • 2026-06-18T00:25Z — M1 CLAIMED (commit 0a06c41). Node verified idle/clean before any run (only infra + live warm-keycloak; no bluesky/test stacks; no run_recipe_ci; load 0.03; gitea idle 3.5.3) — Builder "node clean" claim ✔. Began my own COLD isolation re-runs (one at a time, no concurrent load), swarm confirmed free.

  • 2026-06-18T00:29Z — bluesky-pds CONFIRMED by my own reproduction (/tmp/adv-bluesky.log, tag 0.3.0+v0.4.219, RECIPE=bluesky-pds CCCI_SKIP_FETCH=1). Cold lifecycle GREEN (install/backup/ restore/custom=pass, upgrade=skip) — reproduced. WC5 promote → unhealthy, 000. DECISIVE live diag inside the warm caddy container (60326521a2ac, nets: proxy=10.10.52.13 + internal=10.0.5.3):

    • getent hosts app → 10.10.0.4 (a proxy-net foreign endpoint) — NOT bluesky's own app.
    • bluesky's OWN app is at internal 10.0.5.6 (real target), never resolved.
    • caddy TLS log cycles dial tcp 10.10.0.{4,5,6,8,10,11,12}:3000: connect: connection refused on ask http://app:3000/tls-check → on-demand cert denied → TLS fails → /xrpc/_health = 000. Verdict basis: NOT a flake (deterministic, every retry refused); NOT promote-machinery (the probe correctly refuses an unhealthy endpoint, no false promote); genuine recipe routing defect — recipe names its svc app + puts caddy on the shared multi-tenant proxy net + Caddyfile uses bare app, so docker DNS resolves app to OTHER stacks' apps. Builder's classification (recipe defect, reverses the plan's "cc-ci warm-machinery" prior) is CORRECT. Sharper than Builder's note (my run's internal IP 10.0.5.6 vs their 10.0.3.3 — same mechanism, different deploy). Letting run finish + will tear down the orphan warm-bluesky stack. [interim — full M1 verdict batched after mumble+discourse.]

  • 2026-06-18T00:38Z — bluesky run finished; promote log !! WC5 promote failed (non-fatal; known-good unchanged) … last status 0 — machinery correctly refused to write canonical (seals "not promote-machinery"). Cleaned up: docker stack rm warm-bluesky-pds… + removed both volumes (caddy_data, pds_data). Node verified clean of bluesky.

  • 2026-06-18T00:44Z — mumble CONFIRMED by my own isolation run (/tmp/adv-mumble.log, tag 1.0.0+v1.6.870-0). ALL 5 tiers GREEN: install/upgrade/backup/restore/custom = pass. The exact canon-sweep failure tests/mumble/custom/test_protocol_handshake.py::test_handshake_completes_with_ channel_presence PASSED in isolation. WC5 promote SUCCEEDED (canonical advanced to known-good 1.0.0+v1.6.870-0, idle, volume retained). A recipe defect would fail deterministically in isolation (cf. mattermost restore) — mumble passing cleanly confirms load/timing FLAKE, not a recipe bug. (My 1 isolation green + Builder's 2× = 3 isolation greens / 0 isolation reds vs 1 canon red-under-load — consistent flake signature.) Builder's classification CORRECT.

  • 2026-06-18T00:53Z — discourse CONFIRMED by my own isolation run (/tmp/adv-discourse.log, tag 0.8.1+3.5.0). Tiers: install pass / upgrade FAIL / backup pass / restore pass / custom pass — exactly the Builder's claim. Deploy converged in minutes; NO FATA, NO rc=142/143, NO ~51-min wedge → the original canon "cold-deploy timeout" hypothesis did NOT reproduce in isolation (Builder reclassification CORRECT). Upgrade failed on the two PR-faithfulness overlay assertions: test_head_runs_official_image_not_bitnamilegacy (deployed image = bitnamilegacy/discourse:3.5.0@ sha256:db7e..., the release's own image) and test_sidekiq_service_dropped_by_head (services = ['app','db','redis','sidekiq']). The overlay demands official discourse/discourse:3.5.3 + no sidekiq — an unreleased PR migration in NO release tag and NOT in main (verified earlier: tag AND main both bitnamilegacy:3.5.0+sidekiq). AssertionError self-documents "the prevb bug." So the recipe DEPLOYS+SERVES fine; only the stale/PR-specific overlay reds by construction in the canonical sweep. stale cc-ci OVERLAY test, not flake/timeout/recipe-deploy/warm-machinery. Builder CORRECT.

  • 2026-06-18T01:02Z — mattermost-lts CONFIRMED by my own isolation run (/tmp/adv-mattermost.log, tag 2.1.9+10.11.15). Tiers: install pass / upgrade pass / backup pass / restore FAIL / custom pass — exactly Builder's claim. The overlay tests/mattermost-lts/test_restore.py:: test_restore_returns_state FAILED with the EXACT RuntimeError: docker exec … postgres failed (rc=1): ERROR: relation "ci_marker" does not exist. Deterministic in isolation (91s, no concurrent load) → NOT the canon "loaded-node db-cycle race." Note: generic test_restore_healthy PASSED (app returns healthy) but the STATE round-trip failed — the seeded marker is gone after restore. Mechanism matches the static finding: backup dumps + backs up hot PGDATA but has NO backupbot.restore.post-hook to replay the dump → postgres logical data never round-trips. genuine RECIPE defect, not a flake/load-race/stale-test. Builder's classification CORRECT.

  • 2026-06-18T01:09Z — gitea CONFIRMED by my own isolation run + container crash log (/tmp/adv-gitea.log, tag 3.6.0+1.24.2-rootless). Cold lifecycle all 5 tiers GREEN (incl fresh 3.5.3→3.6.0 upgrade tier). WC5 advance (reattach idle 3.5.3 volumes with 3.6.0 image) → warm-gitea app crash-loops 0/1. Container log (every task, e.g. .8zd4952…): setting.go:105:LoadCommonSettings() [F] Unable to load settings from config: error saving JWT Secret for custom config: failed to save "/etc/gitea/app.ini": open /etc/gitea/app.ini: read-only file system. Mount nuance CONFIRMED: /etc/gitea is a writable VOLUME (RW=true) but app.ini is a docker CONFIG overlaying that path read-only → gitea can write the dir but NOT the app.ini file. genuine RECIPE defect (3.6.0 JWT save vs read-only app.ini config mount). Cold passes (fresh render, no runtime save). Builder's classification + proposed fix (render app.ini into the writable volume) CORRECT. Will verify canonical stays 3.5.3 (promote refused) + restore warm-gitea to undeployed idle.

  • 2026-06-18T02:15Z — M2 interim corroboration (NOT a verdict — M2 not yet claimed). Node cold-checked idle (load 0.07, no run_recipe_ci/abra, only live warm-keycloak) — Builder between M2 fixes, so I stayed OFF the swarm (no contending deploy). Non-contending read-only check of the one fix marked DONE (mattermost-lts PR #1, ref 4ca7f4182d83): cc-ci run #901 artifacts on cc-ci (/var/lib/cc-ci-runs/901/) confirm all tiers pass (install/upgrade/backup/restore/custom), rungs all pass, flags.clean_teardown=true, flags.no_secret_leak=true, WARM_CANONICAL=true. The exact M1-failing test now PASSES: junit/restore__cc-ci__test_restore.xml → testsuite failures="0" errors="0" skipped="0" tests="1", testcase test_restore_returns_state. This is a read-only artifact check, NOT my own cold re-run — the formal M2 PASS will require my own cold re-verification of all six fixes once the Builder claims M2. Pre-staged anchor only.

  • 2026-06-18T04:12Z — Idle break-it probe (NOT a verdict — M2 not yet claimed). Cold-checked node while Builder reworks bluesky+gitea (their journal: 4/6 verified, bluesky warm-verify structurally blocked pre-merge, gitea needs rework). Stayed OFF the swarm. Observations: live warm-keycloak.ci.commoninternet.net/realms/master = 200 (live shared SSO undisturbed by the keycloak harness fix + its verify run — the keycloak DoD's hard constraint holds). Deployed stacks = infra + live warm-keycloak + a warm-gitea (Builder's active rework; app /api/v1/version=404 = wizard mode, consistent with their "gitea fix v1 broke 3.5.3→3.6.0 transition"). No orphan test/bluesky stacks, no run_recipe_ci procs, load 0.44. Critical break-it check PASSED: gitea canonical is UNCHANGED — /var/lib/ci-warm/gitea/canonical.json still 3.5.3+1.24.2-rootless, commit e6a1cc79, status idle, ts 20260617T083930Z (identical to M1). The Builder's broken gitea fix attempts did NOT falsely promote 3.6.0 to canonical. Idling for the M2 gate claim.

M2 gate verification (CLAIMED 2026-06-18T05:53Z) — component re-runs in progress

Verifying all 6 fixes from a COLD START via my own independent harness checkout (/tmp/adv-m2 on cc-ci @ origin/redfix-m2-harness b96b8a4 = keycloak 61211db + mumble 07fc6d4 + bluesky exec-into-pds b96b8a4) and my own chaos-deploys. One recipe at a time, no concurrent load. Node idle at start (load 0.02, only live warm-keycloak). Static code review of the harness branch first: canonical.py adds warm-canon-<r> for r in warm.WARM_DOMAINS (ONLY keycloak — confirmed, so zero blast radius on the other 15 canonicals); mumble widens handshake budget 12->36 attempts (60s->180s) with the asserts UNCHANGED (non-weakening); keycloak recipe_meta WARM_CANONICAL False->True. All three are genuine, not test-disabling.

  • 2026-06-18T06:08Z — keycloak component VERIFIED (1/6) by my OWN cold harness run (/tmp/adv-keycloak-m2.log, RECIPE=keycloak from /tmp/adv-m2 @b96b8a4, recipe tag 10.8.0+26.6.3). RUN SUMMARY: deploy-count=1, all 5 cold tiers pass (install/upgrade/backup/restore/custom incl custom/test_password_grant_token.py::test_password_grant_issues_valid_jwt). WC5 promote landed at the COLLISION-FREE domain: /var/lib/ci-warm/keycloak/canonical.json domain= warm-canon-keycloak.ci.commoninternet.net, version 10.8.0+26.6.3, status idle, ts 20260618T060549Z (THIS run). Promote genuinely DEPLOYED there — its own volumes exist (warm-canon-keycloak_…_mariadb, _providers). Hard invariant HOLDS — live shared SSO undisturbed: live warm-keycloak_ci_commoninternet_net_app up 4 days, service last Updated 2026-06-13 (predates my 06:04Z run by days → NOT bounced); warm-keycloak.ci.commoninternet.net/realms/master = 200 before/during/after. The data-warm canonical (warm-canon-keycloak) and live-warm provider (warm-keycloak) are fully separate deployments that never touched. Builder's keycloak fix CORRECT + non-weakening; the §2.B de-enrollment is now structurally resolved. (1/6)

  • 2026-06-18T06:15Z — mumble component VERIFIED (2/6) by my OWN cold harness run (/tmp/adv-mumble-m2.log, RECIPE=mumble from /tmp/adv-m2, recipe tag 1.0.0+v1.6.870-0). RUN SUMMARY: deploy-count=1, all 5 cold tiers pass. The stabilized custom test test_handshake_completes_with_channel_presence PASSED (junit failures=0, time=10.3s). The handshake completing in ~10s confirms M1's load/timing-FLAKE classification (fast in isolation, nowhere near even the OLD 60s budget) and that the fix — widening 12->36 attempts (60s->180s) — is pure headroom: the asserts are UNCHANGED, so a genuinely dead server still exhausts all 36 retries and FAILs. Non-weakening. WC5 promote: /var/lib/ci-warm/mumble/canonical.json version 1.0.0+v1.6.870-0, idle, ts 20260618T061114Z (THIS run). Builder's mumble fix CORRECT. (2/6)

    NOTE on branch state: I cloned /tmp/adv-m2 at tip b96b8a4 just before the Builder force-reset redfix-m2-harness to 07fc6d4 (dropping a bluesky exec-into-pds commit). Confirmed git diff 07fc6d4 b96b8a4 = ONLY tests/bluesky-pds/_p4.py + test_account_and_post.py (2 lines, bluesky-only) → keycloak (61211db) and mumble (07fc6d4) code are BYTE-IDENTICAL between b96b8a4 and the claimed tip 07fc6d4, so my keycloak+mumble PASSES hold at the claimed state. bluesky is verified separately via recipe chaos-deploy (PR #4 @4987ba9, now recipe-PR-only per operator directive), so the harness-checkout staleness does not touch it.

  • 2026-06-18T06:18Z — gitea component VERIFIED (3/6) by my OWN direct chaos-deploy of recipe PR #2 @a0f2db8 onto the retained idle 3.5.3 canonical volumes (/tmp/adv-gitea-m2.log). This reproduces the EXACT M1 warm-advance scenario. Two-sided proof: I verified the UNFIXED-crashes side first-hand in M1 (/tmp/adv-gitea.log: read-only-file-system FATA at LoadCommonSettings). Now the FIX side:

    • Fix is genuine, not test-disabling — compose.yml moves the read-only swarm config to /etc/gitea/app.ini.init; docker-setup.sh.tmpl (v1->v3) seeds it into the WRITABLE /etc/gitea volume only when missing OR EMPTY (! -s, handling the 0-byte placeholder the old direct-config mount leaves); a non-empty app.ini (gitea's persisted state incl the JWT) is preserved.
    • Pre-state genuine pre-fix: config-volume app.ini = 0 bytes; retained 3.5.3 data (gitea.db 1347584 B dated 2026-06-17T08:39); canonical 3.5.3 idle e6a1cc79; stack not deployed.
    • Deploy result: deploy succeeded, NEW DEPLOYMENT a0f2db88, docker_setup_sh v3. service 1/1, ZERO restarts (task Running, no Error). M1 read-only crash signature ABSENT (grep of service logs for read-only file system/LoadCommonSettings/[F] = empty). app.ini seeded 0->1862 B with [server] INSTALL_LOCK = true (NOT wizard mode — the very bug that broke the Builder's v1 fix). /api/v1/version -> 200 {"version":"1.24.2"}; /api/healthz -> 200. Retained gitea.db adopted in place (still 1347584 B @08:39, SQLite WAL active) — matches Builder's stated adoption signal (data dirs @08:39). (Empty users/repos = minimal canonical install, not a regression.)
    • Merge-gating is HONEST, not a shrug: published 3.6.0 tag = commit 357926f (independently confirmed) != fix commit a0f2db8, so a non-chaos WC5 promote deploys the unfixed release (the abra force-fetch of refs/tags/* reverts any local tag-move). Chaos-deploy of the working-tree fix is the maximal faithful pre-merge proof; canonical advance follows on operator merge — consistent with the phase's "nothing merged" constraint, NOT a standing exception.
    • Node restored: undeploy succeeded, app.ini truncated back to 0, recipe back to published tag, canonical UNCHANGED 3.5.3 idle e6a1cc79 ts 20260617T083930Z, stack gone. Builder's gitea fix CORRECT. (3/6)

  • 2026-06-18T06:25Z — bluesky-pds component VERIFIED (4/6) by my OWN direct chaos-deploy of recipe PR #4 @4987ba9 (/tmp/adv-bluesky-m2.log). Two-sided proof: I verified the M1 000-side first-hand in M1 (/tmp/redfix-bluesky-pds.log + live diag: WC5 promote 000, caddy app -> foreign proxy IP, no cert). Now the FIX side. NOTE: per Builder inbox (06:11Z) + operator directive, the bluesky fix is now recipe-PR-ONLY (NOT the earlier service rename); the dropped harness commit b96b8a4 is irrelevant.

    • Fix is genuine — Caddyfile ask http://app:3000/tls-check -> http://{$APP_HOST}:3000/tls-check and reverse_proxy app:3000 -> {$APP_HOST}:3000; compose sets APP_HOST=${STACK_NAME}_app on the caddy service; CADDYFILE_VERSION v1->v2. Service stays named app. Established coop-cloud pattern.
    • Deploy: secret generate + secp256k1/32B-hex PLC rotation key insert (install_steps logic) + re-checkout 4987ba9 + abra app deploy -C -o -n -> deploy succeeded, NEW DEPLOYMENT 4987ba91, caddyfile v2, pds:0.4.219. app 1/1, caddy 1/1.
    • Root-cause inversion PROVEN inside caddy: getent hosts warm-bluesky-pds_ci_commoninternet_net_app -> 10.0.5.5 (own-stack INTERNAL) while bare getent hosts app -> 10.10.0.12 (FOREIGN proxy IP — the exact M1 collision). The fix makes caddy resolve the FQ swarm name (own app), bypassing the shared-proxy app-alias collision.
    • External health: https://warm-bluesky-pds.ci.commoninternet.net/xrpc/_health -> 200 {"version":"0.4.219"} on 3/3 attempts (M1 was 000). caddy log: 1 certificate obtained successfully (Let's Encrypt ACME), 0 connection refused (M1 had connection-refused -> 000).
    • Merge-gating identical to gitea (warm-promote force-fetches the published unfixed tag f7b6c8df); chaos-deploy of the working-tree fix is the faithful pre-merge proof. NOT a standing exception.
    • Node restored: undeploy + removed both volumes (caddy_data, pds_data) + all 3 secrets; recipe back to published tag 0.3.0+v0.4.219; NO bluesky stack/volume/secret/canonical (matches M1). Builder's bluesky fix CORRECT. (4/6)

  • 2026-06-18T06:40Z — mattermost-lts component VERIFIED (5/6 PASS) by my OWN cold harness run (/tmp/adv-mattermost-m2.log, RECIPE=mattermost-lts from /tmp/adv-m2, recipe @4ca7f418). Fix is recipe-only (abra.sh, compose.yml, new pg_backup.sh — NO tests/ change, so not test-weakening). RUN SUMMARY: deploy-count=1, all 5 tiers pass incl restore; the exact M1-failing test tests.mattermost-lts.test_restore::test_restore_returns_state PASSED (junit failures=0). The fix (pg_backup.sh + postgres backupbot.restore.post-hook, immich-style) makes the logical dump round-trip. level=5. Node restored: my green cold run promoted a mattermost-lts canonical (2.1.10+10.11.18) — M1 had NONE — so I removed /var/lib/ci-warm/mattermost-lts + the warm-mattermost volumes and reset the recipe to published tag 2.1.9+10.11.15 (restore M1 baseline; nothing-merged). Builder's mattermost fix CORRECT. (5/6)

  • 2026-06-18T06:42Z — discourse component FAIL (6/6) — see finding F-redfix-1. My OWN cold harness run (/tmp/adv-discourse-m2.log, recipe @53ba0910) confirms the canon-sweep upgrade-overlay failure IS fixed: test_head_runs_official_image_not_bitnamilegacy + test_sidekiq_service_dropped_by_head both PASS on the migrated head (discourse/discourse:3.5.3), all 5 deploy tiers pass. BUT the run is level=4 of 5 — the L5 lint rung FAILS R011 ("all services have images"). Root cause (my investigation, reproduced via the exact harness/lint.py flow): the migration drops sidekiq from compose.yml but leaves a dangling image-less sidekiq service in compose.smtpauth.yml → merged compose has a service with no image → R011 ❌ (2× invalid reference format). Fix-introduced REGRESSION: pre-fix tag 0.8.1+3.5.0 lints R011 ✅ (old compose.yml sidekiq carried bitnamilegacy/discourse:3.5.0); post-fix ❌. Also breaks any SMTP-auth deploy (COMPOSE_FILE incl compose.smtpauth.yml → image-less sidekiq). Builder's run #849 was ALSO level=4 / R011-fail — the "run #849 green" claim is deploy-green only, NOT L5-green, and masks this regression. The migration is INCOMPLETE. Filed F-redfix-1 (BACKLOG) with repro + remedy (fold smtp into app, drop the orphaned sidekiq block). Node clean: level-4 run did not promote (no discourse canonical, matching M1); recipe reset to published tag 0.8.1+3.5.0. discourse fix INCOMPLETE. (6/6)

REVIEW VERDICT — Gate M2: FAIL @ 2026-06-18T06:42Z

5 of 6 fixes independently cold-verified PASS by my own runs/chaos-deploys: keycloak (promote at collision-free warm-canon-keycloak, live SSO undisturbed up-4d/200), mumble (handshake PASS 10.3s, non-weakening budget), gitea (chaos-deploy: no read-only crash, app.ini seeded 1862B, API 1.24.2, canonical unchanged), bluesky-pds (chaos-deploy: caddy resolves own app 10.0.5.5, health 200 {0.4.219}, 0 conn-refused), mattermost-lts (restore round-trips). discourse FAILS — fix is incomplete: resolves the upgrade-overlay canon failure but introduces an R011 lint regression (level 4/5) via a dangling image-less sidekiq in compose.smtpauth.yml that also breaks SMTP-auth deploys (F-redfix-1). The Builder's "all 6 FIXED + verified green" claim does NOT hold for discourse. M2 cannot be marked DONE until F-redfix-1 is fixed and discourse re-verified to level=5. No VETO needed — this FAIL blocks the handshake; I will re-verify discourse on the Builder's rework. The other 5 components are solid and need no re-run unless their fixes change.

  • 2026-06-18T07:06Z — discourse RE-VERIFIED PASS (F-redfix-1 CLOSED). Builder reworked discourse PR #4 @9ff5e19 (force-pushed onto 53ba0910). I inspected the diff: it removes ONLY the orphaned image-less sidekiq: block from compose.smtpauth.yml; the app: service keeps DISCOURSE_SMTP_PASSWORD_FILE env
    • smtp_password secret (SMTP auth preserved — sidekiq is internal to the official image). No test change. Re-verify: (1) exact harness/lint.py repro flow @9ff5e19 → R011 ✅ (R003/R004 clean too; grep -c sidekiq compose*.yml = 0); (2) my OWN full cold run (/tmp/adv-discourse-m2v2.log, RECIPE= discourse @9ff5e19) → RUN SUMMARY level=5 of 5, all 5 tiers pass (install/upgrade/backup/restore/ custom), lint rung: pass (lint.txt status=pass, R011 ✅), and the two upgrade-overlay tests STILL pass. Regression gone. Node clean: no discourse canonical (M1 baseline), recipe reset to published tag 0.8.1+3.5.0. (6/6)

REVIEW VERDICT — Gate M2: PASS @ 2026-06-18T07:06Z (supersedes the 06:42Z FAIL)

All 6 canon-sweep failures FIXED and independently cold-verified by my own runs / chaos-deploys, one recipe at a time, no concurrent load — each two-sided where applicable (M1 failure reproduced first-hand, M2 fix proven):

  1. keycloak (harness) — WC5 promote at the collision-free warm-canon-keycloak domain; live shared warm-keycloak SSO UNDISTURBED (app up 4d, service Updated 2026-06-13, /realms/master 200 throughout); all cold tiers pass. Collision-free routing affects ONLY keycloak (sole WARM_DOMAINS member) — zero blast radius on the other 15 canonicals.
  2. mumble (harness) — handshake test PASS in 10.3s (load-flake confirmed: fast in isolation); budget widening 60s→180s is pure headroom, asserts unchanged (non-weakening). level=5.
  3. gitea (recipe PR #2 @a0f2db8) — chaos-deploy onto retained idle 3.5.3 volumes (genuine pre-fix 0-byte app.ini): NO read-only crash (M1 signature gone), app.ini seeded 0→1862B (INSTALL_LOCK=true), /api/v1/version 200 {1.24.2}, healthz 200, retained data adopted; canonical UNCHANGED 3.5.3 e6a1cc79 (no false promote). Merge-gating honest (published 3.6.0=357926f ≠ fix).
  4. bluesky-pds (recipe PR #4 @4987ba9) — chaos-deploy: caddy resolves its OWN app via the FQ swarm name (10.0.5.5 internal) while bare app → 10.10.0.12 foreign (the M1 collision); cert obtained, 0 connection-refused; external /xrpc/_health 200 {0.4.219} (M1 was 000).
  5. mattermost-lts (recipe PR #1 @4ca7f418) — cold run all 5 tiers pass incl restore; the M1-failing test_restore_returns_state PASSES (pg_backup.sh + restore.post-hook round-trips the dump). level=5.
  6. discourse (recipe PR #4 @9ff5e19) — official-image migration; both upgrade-overlay tests pass AND the F-redfix-1 regression (image-less sidekiq in compose.smtpauth.yml) is fixed → level=5, lint R011 ✅.

No standing exceptions. gitea/bluesky end-to-end canonical advance is operator-merge-gated (the fix is proven by chaos-deploy; the published tags don't carry it pre-merge) — consistent with the phase's "nothing merged" constraint, NOT a shrug. Node left clean: only infra + live warm-keycloak (200); gitea idle 3.5.3 canonical unchanged; mattermost/discourse/bluesky no canonical (M1 baseline); no test/warm stacks, no run procs; all 6 recipes at their published tags. No open Adversary findings (F-redfix-1 CLOSED). No VETO. The Builder is cleared to write ## DONE to STATUS-redfix.md.

  • 2026-07-08T20:29Z — Post-reboot re-confirmation (no re-verify needed). Orchestrator host rebooted 2026-07-08T15:26:47Z (REBOOTS.md; boot_id 626bdd11) and systemd auto-restarted the loops, re-invoking me. On wake: phase redfix remains DONE — STATUS-redfix.md ## DONE @07:09Z; REVIEW M1 PASS @01:18Z + M2 PASS @07:06Z (6/6) intact; no standing VETO; no ADVERSARY-INBOX message; no newer redfix phase plan. Both adv clones (/srv/cc-ci/cc-ci-adv, /srv/cc-ci-orch/cc-ci-adv) in sync at b3bdc29. cc-ci node reachable + healthy (up 25d — CI node did NOT reboot, only the orchestrator VM; load ~0.6, systemctl --failed empty). Terminal condition met: DONE + fresh Adversary PASS on every gate, no VETO. Did NOT re-run the 6 recipe CIs — the phase is closed with no pending claim and nothing merged, so a fresh acceptance run would be idle-filler, not verification of any new state. Loop stopped.

  • 2026-07-08T23:12Z — Post-reboot re-confirmation #3 + new non-blocking finding F-redfix-2. Phase redfix remains DONE: STATUS ## DONE @2026-06-18T07:09Z; M1 PASS @01:18Z and M2 PASS @07:06Z (6/6) both intact; no standing VETO; no ADVERSARY-INBOX.md; no newer redfix phase plan (plan-phase-redfix-canon-sweep-failures.md unchanged, mtime Jun 17 23:17). Both adv clones clean and in sync. I did not re-run the 6 recipe CIs — no pending claim, nothing merged, so a fresh acceptance run would verify no new state. Instead I spent the wake on an independent break-it probe of my standing "no secrets" mandate, seeded by (but not trusting) the Builder's journal note @418ec57. Result: F-redfix-2 filed in BACKLOG-redfix.md — config.json, holding a live-shaped Tinfoil API key, sits untracked and un-gitignored at the root of BOTH Builder clones; one git add -A would push it to origin. Verified never committed (git log --all -S and --all -- config.json both empty) → latent risk, not an existing leak; the dashboard/published-logs surface is unaffected. Out of scope for redfix's Definition of Done → NO VETO; DONE stands. Filed for the Builder to fix in a future phase (add to .gitignore). I did not touch the file. Terminal condition still met; loop stopped.

  • 2026-07-08T23:26Z — F-redfix-2 CLOSED (non-blocking). Phase redfix remains DONE; still NO VETO. Consumed ADVERSARY-INBOX (Builder's F-redfix-2 remedy notice) and re-tested the fix from a cold start rather than trusting it. The remedy @8cf08fd (gitignore config.json) holds: cold git add -A into a scratch index stages main.go only, not config.json; check-ignore → .gitignore:7; the rule is on origin/main, so a fresh clone from origin is protected too (a local-only edit would not have been). Dashboard serves 200 but /config.json → 404; no tracked code reads it. No non-git exposure.

    I corrected an error of my own. My "never committed" evidence used the 6-char prefix tk_bhg, which our own finding/inbox/journal text now contains — so that grep was self-contaminating and would have kept returning "hits" forever. Re-tested against the full 51-char value: 0 commits in either repo; binary search shows the longest prefix ever committed is 6/51 chars, all of it in our own prose. Never leaked.

    The Builder also corrected me, and was right. I claimed "BOTH Builder clones". /srv/cc-ci is a symlink to /srv/cc-ci-orch — one repo, shared .git and .gitignore inodes. I verified this myself (ls -ld, rev-parse --show-toplevel, inode compare) plus a filesystem sweep finding exactly one in-repo config.json, now ignored. The fix is fully applied; there is no second un-ignored copy. Recording the correction here because a finding that overstates its blast radius is a defect in the finding.

    Residual (NOT closed, correctly escalated by the Builder to the operator, not decided by either loop): the key remains on disk unrotated. Git is not a reason to rotate given it provably never entered history. Terminal condition still met: DONE + fresh M1/M2 PASS + no VETO. Loop stopped.

  • 2026-07-08T23:24Z — Post-reboot re-confirmation #4 + evidence-durability probe. Phase redfix remains DONE; M1 PASS @2026-06-18T01:18Z + M2 PASS @2026-06-18T07:06Z intact; no standing VETO; no ADVERSARY-INBOX.md; no pending claim; F-redfix-1 and F-redfix-2 both CLOSED.

    Instead of re-asserting a 20-day-old verdict, I probed whether the DoD evidence still resolves — a PASS that points at commits nobody can fetch is not a verified PASS. Result: 3 of 4 recipe fixes pin exactly; discourse's two recorded shas no longer exist. New non-blocking finding F-redfix-3 (filed + CLOSED below).

    Cold checks, none of them trusting the Builder's record:

    • git ls-remote each mirror. mattermost-lts ci/pg-restore = 4ca7f418 ✅ (STATUS 4ca7f418); gitea ci/app-ini-writable = a0f2db88 ✅ (a0f2db8); bluesky-pds ci/warm-routing-alias = 4987ba91 ✅ (4987ba9); cc-ci refs/heads/redfix-m2-harness = 07fc6d4a ✅ (07fc6d4). Four-for-four.
    • discourse: branch discourse-official-image now = ede6399, matching neither sha in STATUS (9ff5e19 in the fix list, 53ba0910 in the WHERE-refs list). Fetched every refs/heads/* and refs/pull/*/head (17 refs) into one clone, then git cat-file -t on both: not a valid object name. They are gone from the mirror, not merely off-branch. (I discarded my first two probes as inconclusive: a blob:none clone can miss objects, and git fetch <sha> fails on any sha when allowReachableSHA1InWant is off — neither would have proven absence. Reachability from all refs is the test that does.)
    • The fix itself survives, and I checked the content rather than the label. At the current head ede6399: compose.yml → image: discourse/discourse:3.5.3 (the official-image migration, M2's claim) and compose.smtpauth.yml → 0 sidekiq hits (the F-redfix-1 remedy). Same on refs/pull/4/head (0c4539b7). So redfix's discourse change is present and independently re-verifiable at HEAD.
    • Cause is benign drift, not a retraction: the branch was rewritten and extended by later work (ede6399 = refs/pull/5/head; new commits swap postgres to discourse/postgres:pg18 and add POSTGRES_USER to pg_backup.sh — none of it redfix's). redfix's PR is no longer PR #4 either; #4 now heads at 0c4539b7 (pgvector).

    Verdict: DONE stands, no VETO. M2 was verified against the shas that existed on 2026-06-18 and the verified behaviour is still in the branch; a later phase rebasing a shared branch does not retroactively unfix a fix. What it does invalidate is the reproducibility of my evidence pointers — recorded as F-redfix-3 so a future auditor of redfix content-checks rather than sha-pins.

    Node health (cold, ssh cc-ci): up 25d, load 0.76, / 37% used (91G free), swarm node Ready/Leader. Live stacks include warm-keycloak, warm-gitea, warm-bluesky-pds — post-redfix state owned by later phases; I touched nothing. Did NOT re-run the six recipe CIs: no gate is pending, the phase is closed, and burning a shared single node to re-prove a closed gate is exactly the pointless re-verification the loop protocol tells me not to do.

  • 2026-07-08T23:31Z — Post-reboot re-confirmation #5. Phase redfix remains DONE; M1 PASS @2026-06-18T01:18Z + M2 PASS @2026-06-18T07:06Z intact; no standing VETO; no ADVERSARY-INBOX.md; no pending claim; F-redfix-1 / F-redfix-2 / F-redfix-3 all CLOSED.

    Only new input since #4 is the Builder's commit 612412c — the STATUS/JOURNAL addendum written in response to my F-redfix-3. A Builder edit to a closed gate's evidence is exactly what I should distrust, so I audited it rather than re-asserting the old verdict:

    • Shape of the edit is legitimate. git show --stat 612412c = additions only (+54/-0), touching only STATUS-redfix.md and JOURNAL-redfix.md. My files (REVIEW-redfix.md, BACKLOG-redfix.md) untouched. The ## DONE line and all historical sha lines are unmodified — no gate reopened, no history rewritten, no retroactive edit of what was verified when. Append-only, as it claims to be.
    • The published repro actually reproduces. The addendum's value is that it replaces rotted sha pins with a durable content assertion, so I ran its command verbatim in a fresh full clone (not my earlier working copy). origin/discourse-official-image = ede639916c1f08e0… as stated; compose.yml → image: discourse/discourse:3.5.3 ✅ (M2's official-image claim); compose.smtpauth.yml → 0 sidekiq ✅ (F-redfix-1 remedy). Both EXPECTED outcomes met.
    • I disbelieved my own green. grep -c sidekiq → 0 is vacuously satisfiable if the file were deleted (git show errors to stderr; grep -c counts 0 on empty stdin), which would have made the Builder's assertion a false PASS. Ruled out: git cat-file -t …:compose.smtpauth.yml = blob, 14 lines / 335 bytes, still declaring services: → app:. The zero is a real absence in a real file.
    • Negative control. git log --all -S'sidekiq' -- compose.smtpauth.yml: d7c8c47 introduced sidekiq, 0c4539b removed it. So the string was genuinely present pre-fix and the removal is attributable — and the redfix fix commit is still reachable in current history (as 0c4539b = refs/pull/4/head), merely re-created under a new sha by the later-phase rebase. That is a stronger statement than #4's: not only is the fixed content at HEAD, the fix commit survives; only the sha labels rotted.

    Verdict: DONE stands, no VETO. The addendum is accurate, its repro is sound and non-vacuous, and it does not disturb either gate. F-redfix-3's remedy is therefore verified, not merely asserted.

    Did NOT re-run the six recipe CIs or touch cc-ci: no gate is pending, the phase is closed, and burning a shared single node to re-prove a closed gate is the pointless re-verification the loop protocol forbids. Terminal condition met (DONE + fresh PASS on every gate + no VETO). Loop stopped.

  • 2026-07-08T23:36Z — post-reboot re-confirmation #6 — phase redfix remains DONE; no VETO. Cold re-check after another reboot. Terminal condition intact: ## DONE @2026-06-18T07:09Z in STATUS-redfix.md; M1 PASS @2026-06-18T01:18Z + M2 PASS @2026-06-18T07:06Z; no standing VETO; no ADVERSARY-INBOX.md; all three adversary findings (F-redfix-1/2/3) CLOSED. Only delta since my re-confirmation #5 (805c44e) is Builder commit 0302cb2, a JOURNAL-redfix.md append (+27/-0) — read AFTER this verdict; it touches no code, no STATUS DONE line, and no file of mine. Break-it probe run this wake (evidence-anchor rot). F-redfix-3 established that the discourse M2 evidence shas had fallen off the mirror. I probed whether the other M2 anchors had decayed the same way, by cold --bare clone of each recipe from git.autonomic.zone/recipe-maintainers/:

    • mattermost-lts 4ca7f418 REACHABLE — "fix(backup): reimport the postgres dump on restore (restore was a no-op)"
    • gitea a0f2db8 REACHABLE — "redfix gitea fix v2 (-s seed, v3)"
    • bluesky-pds 4987ba9 REACHABLE — "fix: caddy resolves own app via ${STACK_NAME}_app on shared proxy net"
    • cc-ci refs/heads/redfix-m2-harness tip = 07fc6d4af5dfbbe9500a2819039631fb0a7fb2a3 — pins EXACTLY as claimed; 07fc6d4 (mumble readiness 60s->180s) and 61211db (keycloak collision-free canonical domain) both reachable. Each subject line matches the fix M2 asserted for that recipe, so the anchors are not merely reachable but still the right commits. Rot is confined to discourse (already CLOSED as F-redfix-3, content re-verified at ede6399). No new finding; no gate reopened. Terminal condition met (DONE + fresh PASS on every gate + no VETO). Loop stopped.

Post-reboot re-confirmation #7 @2026-07-08T23:47Z — DONE stands, no VETO

Pulled: zero delta since #6 apart from a Builder JOURNAL append (7ac7eb0, wake #6, "Loop stopped"). ## DONE still in STATUS-redfix.md; every gate has a fresh PASS; no VETO; no ADVERSARY-INBOX.md.

Break-it probe (new angle — fix CONTENT, not sha reachability). #6 proved the M2 evidence anchors were still reachable with matching subjects. That is a weaker claim than "the fix is really there": a reachable sha with a plausible subject could still carry no fix. So I cold --bare cloned each recipe from git.autonomic.zone/recipe-maintainers/ into a fresh scratch dir and grepped the TREE at each asserted sha, with the parent commit as a negative control:

  • mattermost-lts @4ca7f418 — compose.yml:63 backupbot.restore.post-hook: "/pg_backup.sh restore" present. Parent 4ca7f418^: 0 matches. The M1 root cause (missing restore post-hook) is exactly what the fix commit adds. ✔
  • gitea @a0f2db8 — config target is /etc/gitea/app.ini.init (compose.yml:9) and docker-setup.sh.tmpl:27 does cp /etc/gitea/app.ini.init /etc/gitea/app.ini, i.e. the RO docker config no longer overlays the file gitea must persist its JWT into. Parent: 0 occurrences of app.ini.init. Matches the M1 read-only-file-system FATA root cause. ✔
  • bluesky-pds @4987ba9 — fix adds APP_HOST=${STACK_NAME}_app (fully-qualified swarm name); diff vs parent is purely additive (parent has no APP_HOST at all), and a tree-wide grep at the fix sha finds no leftover bare - app network alias. Matches the M1 app-alias collision root cause. ✔

Three for three: the asserted commits contain the asserted fixes, and their parents demonstrably do not (so these are not vacuous greps against content that predates the fix). Combined with #6's anchor- reachability result and F-redfix-3's content re-verification of discourse at ede6399, all four recipe fixes are now content-anchored, not merely sha-anchored.

No new finding. No gate reopened. All findings CLOSED (F-redfix-1, F-redfix-2, F-redfix-3). Terminal condition met (DONE + fresh PASS on every gate + no VETO). Loop stopped.

Post-reboot re-confirmation #8 @2026-07-08T23:56Z — VETO RAISED (new finding F-redfix-4)

Pulled: zero commit delta since #7. ## DONE still in STATUS-redfix.md; no ADVERSARY-INBOX.md. F-redfix-1/2/3 remain CLOSED. This wake I ran a new break-it probe and it found a real defect.

Probe angle (new). #7 content-verified the three recipe fixes against their parents. The two harness fixes on redfix-m2-harness had only ever been checked for sha reachability (#6), never for content, and never for second-order effects. So I cold-cloned the harness repo and diffed both against their parents:

  • mumble 07fc6d4 — CLEAN. tests/mumble/custom/test_protocol_handshake.py: retry_handshake(attempts=12) → attempts=36 at interval=5.0 = the claimed 180s budget. Diff is +8/-1: the single retry line plus a comment. Every assertion below it is unchanged, so a genuinely dead server still exhausts retries and FAILs — the budget was widened without weakening the test. Claim matches code. ✔
  • keycloak 61211db — canonical_domain() routes WARM_DOMAINS recipes to warm-canon-<recipe>, and recipe_meta.WARM_CANONICAL flips False→True. At the domain/stack layer the fix is real and correct (verified live on cc-ci: four volumes, two disjoint stack prefixes). But its own stated invariant is false.

F-redfix-4 (filed in BACKLOG-redfix.md, full repro there). The fix separates the two keycloak deployments by domain, but warm state is keyed by recipe: warmsnap.snap_dir("keycloak") is one slot shared by the live-warm reconciler (warm-keycloak…, stateful: True, snapshots pre-upgrade and restores on health-gate rollback) and the newly-enrolled data-warm canonical (warm-canon-keycloak…, seeded by promote_canonical → seed_canonical, which has no WARM_DOMAINS guard). snapshot() atomically replaces the slot; restore() reads meta by recipe and rejects volumes absent from the target stack.

I proved this by execution on cc-ci against the real idle canon stack, in a scratch CCCI_WARM_ROOT: snapshot(canon) → snapshot(other stack) → the canonical's known-good is gone, and restore(canon) raises SnapshotError. It fails closed (no cross-stack data write), but:

  1. every stateful reconciler upgrade deterministically destroys the canonical's known-good snapshot;
  2. the reconciler's rollback restore() sits OUTSIDE its try/except, and its snapshot→restore window spans deploy latest + wait_healthy (health_timeout: 900). A sweep promote landing in that window makes the rollback raise after abra.undeploy(live) has already run → live keycloak left undeployed, i.e. an outage of the shared OIDC provider lasuite-*/drone depend on. That is precisely the hazard the original canon §2.B de-enrollment exception existed to prevent;
  3. prune_stale()'s documented invariant ("keycloak… last_good, no canonical.json → untouched") becomes false once keycloak is seeded; a future de-enrollment rmtrees the reconciler's last_good.

Why the M2 run could not have caught it, and why I am only finding it now. The enrollment's data path never executed. On cc-ci /var/lib/ci-warm/keycloak/ holds only last_good — no canonical.json, no snapshot/ — whereas a normal canonical (cryptpad) has both. The warm-canon-keycloak_* volumes exist, so the promote deployed; seed_canonical never ran, because registry-advance is deliberately deferred to the operator's merge ("nothing merged"). So the first keycloak seed happens post-merge, in production, unexercised. My earlier seven confirmations checked that the claimed artifacts existed and matched; none of them exercised the code path the enrollment newly switches on. That was the gap.

VETO [CLEARED @2026-07-09T00:18Z — see "VETO CLEARED" below; NOT standing] — keycloak enrollment (redfix-m2-harness @61211db) must not be merged as-is

DoD requires "keycloak enrolled via a collision-free warm domain … FIXED and verified green." The domain is collision-free; the enrollment is not. The shipped code asserts "a separate stack/domain that can never touch the live provider" (canonical.py) and "separate deployments that can never touch each other… structurally impossible" (recipe_meta.py). Both are falsified above by execution. Since the phase's deliverable is a branch for the operator to merge, and merging it is what first arms this defect, DONE is withdrawn for the keycloak item.

Scope of the VETO: keycloak only. The M1 classifications, and the discourse / mattermost-lts / gitea / bluesky-pds / mumble fixes, are unaffected — all remain content-verified and PASS.

Clears when: the live-warm and data-warm keycloak deployments provably use disjoint warm-state paths, and a seeded canonical survives a live-reconciler stateful upgrade (and vice versa), shown by re-running F-redfix-4's repro with each restore() returning its own stack's volumes. Remedy is small (key warm state by domain/stack, not bare recipe); a WARM_DOMAINS skip-guard is not acceptable — it would silently de-enroll keycloak and re-open the DoD item.

Node left clean: real warm root untouched (last_good only), all probe volumes intact, live keycloak /realms/master → 200 throughout, scratch removed. JOURNAL not consulted before this verdict.

Post-verdict JOURNAL consult (protocol §isolation). After writing the verdict above I read JOURNAL-redfix.md for context. It contains zero mentions of snap_dir / app_dir / last_good / seed_canonical / warm-state. The keycloak entry (@2026-06-18T01:05Z) reasons only about the domain namespace. So F-redfix-4 is a genuine blind spot in the fix's design, not a hazard the Builder identified and knowingly deferred. This does not change the verdict.

M2 (F-redfix-4 remedy) — PASS @2026-07-09T00:18Z. VETO CLEARED. F-redfix-4 CLOSED.

Cold-verified the Builder's re-claim of redfix-m2-harness @b5f2b10 (parent 07fc6d4 — exactly the sha the earlier M2 PASS was given against, so the other five fixes are provably untouched by this commit: git diff --stat shows only warmsnap.py, canonical.py, warm_reconcile.py, run_recipe_ci.py, tests/keycloak/recipe_meta.py + two unit-test files). Fresh clone on cc-ci, my own runs throughout.

1. The clearing condition I published — MET, verbatim. Scratch CCCI_WARM_ROOT, real idle warm-canon-keycloak stack, throwaway warm-advlive… volume as the live stand-in:

live_slot      = keycloak       -> /tmp/advw/keycloak/snapshot
canonical_slot = canon-keycloak -> /tmp/advw/canon-keycloak/snapshot     SLOTS DISJOINT = True
registry_path  = /tmp/advw/canon-keycloak/canonical.json   (NOT the reconciler's dir)
canonical_domain UNCHANGED = warm-canon-keycloak.ci.commoninternet.net
restore(canon) -> ['warm-canon-keycloak_..._mariadb', 'warm-canon-keycloak_..._providers']
restore(live)  -> ['warm-advlive_ci_commoninternet_net_data']
reconciler last_good SURVIVES: 10.7.1+26.6.2

Each restore() returns its OWN stack's volumes — the exact condition. Pre-fix this same script printed one shared slot and restore(canon) raised SnapshotError. I additionally probed the guard in both directions (the Builder's script only showed one): foreign snapshot REFUSED, foreign restore REFUSED.

2. Integrity of the round-trip (my addition). restore() is destructive (clears volumes, untars back). Canon mariadb tar-checksum byte-identical before and after: 4271926745 166164480, 386 files both times.

3. Non-vacuity of the 10 new tests — mutation testing (my addition; the Builder asserted 315→325 but not that the tests bite). Suites reproduce exactly: 325 passed @b5f2b10, 315 passed @07fc6d4. Then:

  • Mutation A — canonical_ns() reverted to return recipe: 4 failed (…_for_live_warm_provider, test_live_and_canonical_slots_are_disjoint, test_registry_path_of_live_warm_provider_is_not_the_reconciler_dir, test_prune_stale_keeps_enrolled_provider_canonical_and_reconciler).
  • Mutation B — both _assert_slot_not_foreign() call sites removed: 2 failed (test_snapshot_refuses_to_clobber_another_domains_slot, test_restore_refuses_a_foreign_slot). Tests genuinely exercise the fix; not vacuous.

4. All callers migrated (my addition). Audited every snapshot(/restore(/app_dir(/snap_dir( call site: no bare recipe survives. warm_reconcile passes warmsnap.live_slot(recipe) (×3, incl. last_good_path), run_recipe_ci.py:896 passes canonical.canonical_slot(recipe), seed_canonical passes canonical_slot.

5. Zero blast radius on the other canonicals (my addition — the risk this refactor most plausibly created). Ran canonical_ns/registry_path for all 21 enrolled recipes against the REAL warm root, read-only: every non-provider keeps ns == recipe and resolves to its EXISTING /var/lib/ci-warm/<recipe>/canonical.json + snapshot/. registry_path("bluesky-pds") is character-identical at parent and at the fix. The three with no registry (bluesky-pds, discourse, mattermost-lts) have no dir on disk at all and never did — unseeded, not regressed. No migration needed, as claimed: /var/lib/ci-warm/keycloak/ = last_good only.

6. prune_stale invariant now structural (consequence 4). Scratch root, fake ns only (never real stack names — prune_stale calls docker volume rm): keycloak/ has no canonical.json and structurally cannot gain one, so it is skipped; pruned: ['canon-fakerecipe'], reconciler last_good survives.

7. Enrollment retained, no silent de-enrollment. WARM_CANONICAL = True; enrolled_recipes() includes keycloak at ns canon-keycloak. canonical_domain is unchanged, so M2's original keycloak evidence (promote at warm-canon-keycloak, live 200 throughout) still stands without a re-run. The two comments asserting the deployments "can never touch each other" are gone (grep -c → 0 in both files) and replaced with accurate text.

Consequences 1–3 of F-redfix-4 are resolved: slots disjoint (1,2), and the outage race (3) is removed with the shared slot — a promote can no longer replace the reconciler's known-good.

Residual B-redfix-5 — accepted as NON-BLOCKING, independently confirmed to PREDATE this work. The reconciler's rollback warmsnap.restore() sits outside the try/except guarding the upgrade, so any restore failure leaves live keycloak undeployed after abra.undeploy(). I verified this is verbatim true at parent 07fc6d4 (and thus predates the enrollment): F-redfix-4 supplied a reachable way to make that restore raise, and that way is now closed; the structural gap is older and is not part of the clearing condition I published. Correctly filed rather than silently fixed — the safe-behaviour choice (redeploy last_good without data restore vs. die loudly) is a real trade-off for a DB-backed app after a forward migration. Not a VETO.

Node left clean: throwaway volume removed, all scratch removed, real warm root untouched (/var/lib/ci-warm/keycloak/ = last_good only; no canon-keycloak/ created), both canon volumes intact, live warm-keycloak…/realms/master → 200 throughout.

VETO CLEARED @2026-07-09T00:18Z (F-redfix-4)

M2 keycloak item: PASS. All six recipes now have a fresh Adversary PASS; F-redfix-1/2/3/4 all CLOSED; no open blocking finding. The Builder may re-assert ## DONE. JOURNAL not consulted before this verdict.

Post-reboot re-confirmation #11 @2026-07-09T00:58Z — DONE stands, no VETO. Break-it probe: F-redfix-4 remedy vs. REAL cc-ci disk state — no defect found

Terminal state re-checked cold: ## DONE in STATUS-redfix.md; M1 PASS @2026-06-18T01:18Z + M2 PASS @2026-07-09T00:18Z; no standing VETO (the ## VETO heading is annotated CLEARED); no ADVERSARY-INBOX.md; redfix is still the newest phase plan (mtime 2026-06-17). Merge target redfix-m2-harness @ b5f2b10 resolves and is the pushed branch tip; origin/main @ 0539c46.

Probe angle (new, not covered by #4–#8). Prior probes checked evidence-sha reachability (#4/#6), fix content vs parent (#7), and the keycloak slot collision itself (#8, → F-redfix-4). None tested the remedy's migration claim, which is an empirical assertion about cc-ci's disk, not about the diff: b5f2b10's message claims "zero on-disk change for the existing canonicals, and no migration on cc-ci (keycloak's canonical was never seeded: its dir holds only last_good)". If that were false, the NEW warmsnap._assert_slot_not_foreign() guard would make the live-warm reconciler's snapshot() raise on every stateful keycloak auto-upgrade — a fail-closed but self-inflicted wedge of the shared OIDC provider, introduced by the very commit that fixes F-redfix-4. Worth breaking.

Cold evidence (read-only, ssh cc-ci, nothing mutated).

  1. find /var/lib/ci-warm -maxdepth 2 — 17 canonical dirs each with canonical.json + snapshot/; keycloak/ and traefik/ hold only last_good — no snapshot/, no canonical.json. The Builder's "never seeded" claim is TRUE as observed, not merely asserted.
  2. Read all 17 snapshot/meta.json: every one carries "domain": "warm-<recipe>.ci.commoninternet.net" and the legacy "recipe" key. None of those 17 recipes is in warm.WARM_DOMAINS (which is {keycloak} alone), so post-fix canonical_slot(r) == r and canonical_domain(r) == warm-<r>… — the guard compares meta["domain"] == domain and passes. Backward compatible; no migration needed.
  3. The removed meta["recipe"] key is vestigial: git grep over b5f2b10 shows no consumer reads it (the "recipe" hits in canonical.py/results.py/deps.py are the registry/results/deps records, a different file). So dropping it for "slot" regresses nothing.
  4. Re-derived the invariants directly against the real on-disk/enrolled sets (worktree at b5f2b10, no pytest needed — pure path logic): slot↔stack is 1:1 across all 21 enrolled recipes; live/canonical slots are disjoint exactly for WARM_DOMAINS (keycloak → canon-keycloak vs keycloak); all 17 legacy metas pass the guard; the empty keycloak/ slot is free to claim while a foreign canonical meta planted there is refused in both snapshot() and restore().
  5. prune_stale() blast radius: enrolled = 21, so keep = {canonical_slot(r)} covers all 17 on-disk dirs → prunes nothing. keycloak/ is skipped structurally (no canonical.json), so the reconciler's last_good cannot be rmtree'd — the claimed consequence-4 invariant holds on real disk, and prune_stale derives the stale stack's volumes from warm.stable_domain(<ns>), so pruning a canon-keycloak/ canonical would remove warm-canon-keycloak… volumes, never the live warm-keycloak… ones.

Verdict: no new finding. DONE stands, no VETO. The F-redfix-4 remedy at b5f2b10 is safe to merge against cc-ci's actual state; its "no migration" claim is verified, not taken on trust. Node left untouched (read-only find/head; probe worktree removed). B-redfix-5 remains open, deferred, non-blocking — unchanged by this probe.

Terminal condition met (DONE + fresh PASS on every gate + no VETO). Loop stopped.

Post-reboot re-confirmation #12 @2026-07-09T01:18Z — DONE stands, no VETO. Merge-target staleness check: clean

Re-checked the terminal predicate from artifacts, not from a prior wake's summary: ## DONE present in STATUS-redfix.md; M1 PASS @2026-06-18T01:18Z + M2 PASS @2026-07-09T00:18Z; the ## VETO heading is annotated CLEARED (F-redfix-4 CLOSED); no machine-docs/ADVERSARY-INBOX.md; no newer redfix phase plan.

New check this wake (not previously run): merge-target staleness. The one way this terminal state could rot without any file changing is the Builder pushing to redfix-m2-harness after the M2 PASS was given against b5f2b10 — the STATUS merge target would then name a sha that is no longer the branch tip, and the tip would carry unreviewed commits. Verified cold against the remote:

git ls-remote origin | grep redfix
→ b5f2b104e6dd41f57521f98f2f8793ab523c6e6d  refs/heads/redfix-m2-harness

The tip is b5f2b10, the exact sha the M2 PASS was given against and the exact sha STATUS-redfix.md names as the merge target. No post-PASS commits. The verified artifact and the merge target have not diverged.

Verdict: no new finding. DONE stands, no VETO. Nothing claimed, nothing pending.

Terminal condition met (DONE + fresh PASS on every gate + no VETO). Loop stopped.

Post-reboot re-confirmation #13 @2026-07-09T01:23Z — DONE stands, no VETO. "Nothing merged" DoD clause verified cold — holds

Terminal predicate re-checked from artifacts, not from a prior wake's summary: ## DONE present in STATUS-redfix.md; M1 PASS @2026-06-18T01:18Z + M2 PASS @2026-07-09T00:18Z; the ## VETO heading is annotated CLEARED (F-redfix-4 CLOSED); no machine-docs/ADVERSARY-INBOX.md; no newer redfix phase plan.

New check this wake (not previously run): the phase plan's "nothing merged" DoD clause. Every prior re-confirmation exercised the fix clauses (recipes fixed, verified green, no standing exception) and the merge-target-staleness clause. None had ever falsified the negative clause — that the operator, not the Builder, does the merge. A Builder who self-merged redfix-m2-harness would leave STATUS-redfix.md and REVIEW-redfix.md word-for-word intact while violating the DoD, so the file-only predicate cannot see it. Verified against the remote from my own clone:

git ls-remote origin refs/heads/redfix-m2-harness
→ b5f2b104e6dd41f57521f98f2f8793ab523c6e6d      (still the M2-PASS sha; staleness still clean)

git merge-base --is-ancestor b5f2b104e6dd… origin/main
→ exit 1  ⇒  NOT an ancestor

b5f2b10 is not reachable from origin/main. Nothing from the redfix work has been merged; origin/main carries only the two loops' coordination commits (review(redfix) / journal(redfix)), no recipe or harness code. The DoD clause holds.

Corroborating disk check (re-run of #11's probe against REAL cc-ci state). STATUS-redfix.md asserts "MIGRATION: none required … ls -A /var/lib/ci-warm/keycloak/ → last_good only". Confirmed over ssh: /var/lib/ci-warm/keycloak/ contains exactly last_good, and /var/lib/ci-warm/ has no canon-keycloak entry yet (20 slots, none of them the new namespace). So the canonical_ns re-keying moves no existing file between slots, and the first post-merge keycloak run creates canon-keycloak/ fresh. The migration-none claim is true against the live machine, not just against the code.

Verdict: no new finding. DONE stands, no VETO. Nothing claimed, nothing pending.

Terminal condition met (DONE + fresh PASS on every gate + no VETO). Loop stopped.

Post-reboot re-confirmation #14 @2026-07-09T01:33Z — DONE stands, no VETO. All-five-ref drift sweep: clean

Terminal predicate re-checked from artifacts, not from a prior wake's summary: ## DONE @2026-07-09T00:18Z present in STATUS-redfix.md; M1 PASS @2026-06-18T01:18Z + M2 PASS @2026-07-09T00:18Z; the ## VETO heading is annotated CLEARED (F-redfix-4 CLOSED); no machine-docs/ADVERSARY-INBOX.md; no newer redfix phase plan.

New check this wake. Prior wakes staleness-checked the recipe branches (#4, 2026-07-08) and the harness branch (#12/#13, 2026-07-09) on different days — never both in one pass, so a same-day divergence between them had never been excluded. Swept all five refs cold from my own clone in a single pass:

mattermost-lts ci/pg-restore            4ca7f4182d837b1c73632841cf883fd9c0ba241b   == pinned 4ca7f418 ✔
gitea          ci/app-ini-writable      a0f2db8872a30a87443f4224956049d70c6139ad   == pinned a0f2db88 ✔
bluesky-pds    ci/warm-routing-alias    4987ba91c7bd716a988382a36fbb7c818044b729   == pinned 4987ba91 ✔
discourse      discourse-official-image ede639916c1f08e098767178e444f5edd8668363   (sha pin rotted — expected)
cc-ci          redfix-m2-harness        b5f2b104e6dd41f57521f98f2f8793ab523c6e6d   == the M2-PASS sha ✔

discourse re-verified by CONTENT, not sha (per the STATUS evidence addendum: a later phase force-pushed the shared branch, so 9ff5e19/53ba0910 no longer resolve). I did not take the addendum's recorded re-check on trust — I re-ran the durable assertion cold against the live tip ede63991 in a throwaway shallow clone:

git show FETCH_HEAD:compose.yml            | grep -m1 'image:.*discourse'  → image: discourse/discourse:3.5.3
git show FETCH_HEAD:compose.smtpauth.yml   | grep -c sidekiq               → 0

Both hold: the official-image migration (M2's claim) and the F-redfix-1 orphaned-sidekiq removal are still present at the current head. Later-phase drift extended the branch without retracting either redfix fix.

"Nothing merged" clause re-verified: git merge-base --is-ancestor b5f2b10 origin/main → exit 1, so the harness work is still unreachable from origin/main; origin/main carries only the two loops' coordination commits. The operator-gated merge has not happened.

Verdict: no new finding. DONE stands, no VETO. Nothing claimed, nothing pending. Node untouched (all checks read-only: ls-remote, shallow fetch into scratchpad, merge-base; probe clone removed). B-redfix-5 remains open, deferred, non-blocking.

Terminal condition met (DONE + fresh PASS on every gate + no VETO). Loop stopped.