fix(keycloak): key warm state by stack namespace, not bare recipe (F-redfix-4)
Some checks failed
continuous-integration/drone/push Build is failing
Some checks failed
continuous-integration/drone/push Build is failing
The M2 keycloak enrollment made the canonical collision-free at the DOMAIN layer
(warm-canon-keycloak vs warm-keycloak) but warm STATE stayed keyed by bare recipe:
warmsnap.app_dir("keycloak") resolved both the live-warm reconciler's last_good and
the data-warm canonical's canonical.json + snapshot/ into /var/lib/ci-warm/keycloak/.
snapshot() atomically REPLACES that slot, so the two deployments destroyed each
other's known-good; restore() then raised SnapshotError (fails closed, no cross-stack
data write). Worst case: a sweep promote landing inside the reconciler's
snapshot->wait_healthy window makes its rollback restore() raise after
abra.undeploy(live), leaving the shared OIDC provider undeployed.
Fix: canonical.canonical_ns() is now the single namespace from which BOTH the
canonical's domain and its warm-state slot derive, so they cannot drift apart. A
live-warm provider gets ns "canon-<recipe>": domain warm-canon-keycloak (unchanged)
and slot /var/lib/ci-warm/canon-keycloak/. Every other recipe keeps ns "<recipe>" —
zero on-disk change for the 15 existing canonicals, and no migration on cc-ci
(keycloak's canonical was never seeded: its dir holds only last_good).
- warmsnap: functions take a SLOT, not a recipe; add live_slot(); meta records "slot".
- warmsnap: _assert_slot_not_foreign() refuses to snapshot/restore a slot recorded
against a different domain -- defence in depth, naming-scheme-independent, fails
before the destructive swap rather than at the next restore.
- canonical: registry_path/seed_canonical/prune_stale go through canonical_slot().
- prune_stale: the "reconciler dirs are never pruned" invariant is now STRUCTURAL --
<recipe>/ never gains a canonical.json, so de-enrolling keycloak can no longer
rmtree the reconciler's last_good (consequence 4).
- warm_reconcile: last_good + snapshot/restore go through warmsnap.live_slot().
- run_recipe_ci: canonical rollback restores from canonical_slot(recipe).
- Correct the two comments that claimed the deployments "can never touch each other".
Tests: 10 new (slot disjointness for every WARM_DOMAINS recipe, slot<->stack 1:1,
registry not in the reconciler dir, prune spares last_good, foreign-slot refusal in
both snapshot and restore). Unit suite 315 -> 325, no regressions.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FS8p1esg57UAC69riNvuBX
This commit is contained in:
@ -384,7 +384,9 @@ def wait_undeployed(domain: str, timeout: int = 120) -> None:
|
||||
|
||||
|
||||
def last_good_path(recipe: str) -> str:
|
||||
return os.path.join(warmsnap.app_dir(recipe), "last_good")
|
||||
# live_slot, not the bare recipe: a recipe that is also a data-warm canonical keeps its canonical
|
||||
# state in a DIFFERENT slot, so the reconciler's last_good is never clobbered (F-redfix-4).
|
||||
return os.path.join(warmsnap.app_dir(warmsnap.live_slot(recipe)), "last_good")
|
||||
|
||||
|
||||
def read_last_good(recipe: str) -> str | None:
|
||||
@ -396,7 +398,7 @@ def read_last_good(recipe: str) -> str | None:
|
||||
|
||||
|
||||
def write_last_good(recipe: str, version: str) -> None:
|
||||
os.makedirs(warmsnap.app_dir(recipe), exist_ok=True)
|
||||
os.makedirs(warmsnap.app_dir(warmsnap.live_slot(recipe)), exist_ok=True)
|
||||
tmp = last_good_path(recipe) + ".tmp"
|
||||
with open(tmp, "w") as f:
|
||||
f.write(version)
|
||||
@ -509,7 +511,7 @@ def reconcile(app: str) -> str:
|
||||
if stateful:
|
||||
abra.undeploy(domain)
|
||||
wait_undeployed(domain)
|
||||
warmsnap.snapshot(recipe, domain, version=last_good)
|
||||
warmsnap.snapshot(warmsnap.live_slot(recipe), domain, version=last_good)
|
||||
# snapshot requires undeployed; now bring up latest.
|
||||
# A broken "latest" can fail in two ways: deploy_version raises (abra converge times out on a
|
||||
# crash-looping task) OR it deploys but never becomes healthy. BOTH must roll back, so treat a
|
||||
@ -531,7 +533,7 @@ def reconcile(app: str) -> str:
|
||||
if stateful:
|
||||
abra.undeploy(domain)
|
||||
wait_undeployed(domain)
|
||||
warmsnap.restore(recipe, domain)
|
||||
warmsnap.restore(warmsnap.live_slot(recipe), domain)
|
||||
deploy_version(recipe, domain, last_good, dt)
|
||||
recovered = wait_healthy(spec)
|
||||
write_alert(
|
||||
|
||||
Reference in New Issue
Block a user