Cold-verified redfix-m2-harness@b5f2b10 (parent 07fc6d4 = the sha the earlier M2 PASS was
given against, so the other five fixes are provably untouched by this commit).
Clearing condition MET verbatim: live_slot=keycloak vs canonical_slot=canon-keycloak are
disjoint; restore(canon) and restore(live) each return their OWN stack's volumes; reconciler
last_good survives; foreign snapshot AND foreign restore both refused (probed both directions).
canonical_domain unchanged, so M2's original keycloak evidence still stands.
Checks the Builder did not run:
- integrity: canon mariadb byte-identical across the destructive restore round-trip
(cksum 4271926745 166164480, 386 files, before and after).
- mutation testing (are the +10 tests vacuous?): reverting canonical_ns() to `return recipe`
reds 4 of them; removing both _assert_slot_not_foreign() call sites reds 2. Not vacuous.
Suites reproduce exactly: 325 passed @b5f2b10, 315 passed @07fc6d4.
- caller audit: every snapshot/restore/app_dir/snap_dir call site passes an explicit slot;
no bare recipe survives.
- blast radius (the risk this refactor most plausibly created): all 21 enrolled recipes still
resolve to their EXISTING on-disk dirs; registry_path("bluesky-pds") is character-identical
at parent and fix. The 3 without a registry have no dir at all and never did. No migration.
- prune_stale invariant now structural: <recipe>/ never gains a canonical.json, so a
de-enrolled provider can no longer rmtree the reconciler's last_good (scratch-root sim,
fake ns only — prune_stale calls `docker volume rm`).
Enrollment retained (WARM_CANONICAL=True), no silent de-enrollment. Both false
"can never touch each other" comments removed.
B-redfix-5 (reconciler rollback restore() outside the upgrade try/except) accepted as
NON-BLOCKING: verified verbatim present at parent 07fc6d4, so it predates the enrollment.
F-redfix-4 made it reachable; that path is now closed. Correctly filed, not silently fixed.
All six recipes now hold a fresh Adversary PASS; F-redfix-1/2/3/4 CLOSED; no open blocking
finding. Builder may re-assert ## DONE.
Node clean: throwaway volume + scratch removed, real warm root untouched (keycloak/ = last_good
only, no canon-keycloak/ created), canon volumes intact, live keycloak /realms/master 200 throughout.
Post-reboot re-confirmation #8. New break-it probe on the two HARNESS fixes (previously
only sha-reachability checked, never content, never second-order effects).
mumble 07fc6d4: CLEAN — attempts 12->36 @5.0s = claimed 180s; all assertions unchanged,
so a dead server still FAILs. Budget widened without weakening the test.
keycloak 61211db: correct at the domain/stack layer, but its own invariant is FALSE.
canonical_domain() separates the two deployments by domain, yet warm STATE is keyed by
recipe: warmsnap.snap_dir("keycloak") is ONE slot shared by the live-warm reconciler
(warm-keycloak, stateful=True) and the newly-enrolled data-warm canonical
(warm-canon-keycloak, seeded via promote_canonical -> seed_canonical, no WARM_DOMAINS guard).
Proved by execution on cc-ci in a scratch CCCI_WARM_ROOT against the real idle canon stack:
snapshot(canon) -> snapshot(other stack) destroys the canonical known-good, and
restore(canon) raises SnapshotError. Fails closed (no cross-stack data write), but:
1. every stateful reconciler upgrade deterministically destroys the canonical snapshot;
2. the reconciler's rollback restore() is OUTSIDE its try/except and its snapshot->restore
window spans deploy+wait_healthy (health_timeout 900). A sweep promote landing there makes
the rollback raise AFTER abra.undeploy(live) -> live keycloak left undeployed = outage of
the shared OIDC provider lasuite-*/drone depend on. Exactly the hazard canon §2.B's
de-enrollment exception existed to prevent.
3. prune_stale()'s documented "keycloak untouched" invariant becomes false once seeded.
Undetectable by the M2 run: /var/lib/ci-warm/keycloak/ holds only last_good (no canonical.json,
no snapshot/) vs cryptpad which has both — the promote deployed but seed_canonical never ran,
registry-advance being deferred to the operator's merge. First seed happens post-merge, unexercised.
VETO scoped to keycloak. M1 classifications and the discourse/mattermost-lts/gitea/bluesky-pds/
mumble fixes are unaffected and remain PASS. Clears when the two deployments use disjoint
warm-state paths and each restore() returns its own stack's volumes.
Node left clean: real warm root untouched, volumes intact, live keycloak 200 throughout.
