fix(harness): make concurrent recipe runs safe (per-recipe flock + active-run registry)
continuous-integration/drone/push Build is passing
continuous-integration/drone/push Build is passing
capacity=2 went live with three stale capacity=1-era assumptions that corrupted concurrent runs (immich 229/230 '/pg_backup.sh: No such file'): - ~/.abra/recipes/<recipe> is ONE shared working tree that fetch_recipe rm-rf's/ reclones and the upgrade tier git-checkouts mid-run. Same-recipe runs now serialise on an exclusive flock (/run/lock/cc-ci-recipe-<recipe>.lock), taken in main() BEFORE fetch_recipe and held for the whole run; the kernel releases it on any process death, so there is no stale-lock failure mode. Different recipes still run in parallel. - CCCI_JANITOR_MAX_AGE=0 made a starting build reap ANY in-flight run app. Every run now registers its app domain + pid in /run/cc-ci-active/<domain> before app creation; the janitor checks the owner: alive (pid is a live run_recipe_ci process) -> never reaped; dead -> reaped immediately; unknown (pre-registry or post-reboot) -> age fallback (default 2h). The MAX_AGE=0 env override is gone from .drone.yml. - .drone.yml: concurrency.limit 1 -> 2 to match DRONE_RUNNER_CAPACITY=2; the 'safe because capacity=1' comments now describe the flock+registry model. lint: PASS, unit tests: 138 passed.
This commit is contained in:
+10
-8
@@ -35,10 +35,12 @@ steps:
|
||||
# the comment-bridge). Deploys the recipe at the PR head, runs install/upgrade/backup + any
|
||||
# recipe-local tests via the shared harness, then guarantees teardown (plan §4.2/§4.3).
|
||||
#
|
||||
# Resource safety (plan §4.2/§4.3): MAX_TESTS=DRONE_RUNNER_CAPACITY=1 (nix/modules/drone-runner.nix) is
|
||||
# the primary concurrency cap; concurrency.limit below is a redundant belt. CCCI_JANITOR_MAX_AGE=0
|
||||
# makes the run-start janitor reap ANY orphaned run app before deploying — safe because capacity=1
|
||||
# means no concurrent run exists (a SIGKILL'd/timed-out build leaves an orphan with no teardown).
|
||||
# Resource safety (plan §4.2/§4.3): DRONE_RUNNER_CAPACITY=2 (nix/modules/drone-runner.nix) +
|
||||
# concurrency.limit=2 below allow two recipe runs in parallel. Concurrent-run safety is enforced by
|
||||
# the harness, not by serialisation: same-recipe runs serialise on a per-recipe flock
|
||||
# (lifecycle.acquire_recipe_lock — the shared ~/.abra/recipes/<recipe> checkout is the conflict),
|
||||
# and every run registers its app domain + pid in /run/cc-ci-active so the run-start janitor only
|
||||
# reaps orphans whose owning run is DEAD (alive → never touched; unknown → age fallback, default 2h).
|
||||
kind: pipeline
|
||||
type: exec
|
||||
name: recipe-ci
|
||||
@@ -52,16 +54,16 @@ trigger:
|
||||
- custom
|
||||
|
||||
concurrency:
|
||||
limit: 1
|
||||
limit: 2
|
||||
|
||||
steps:
|
||||
- name: ci
|
||||
environment:
|
||||
STAGES: install,upgrade,backup,restore,custom
|
||||
CCCI_JANITOR_MAX_AGE: "0"
|
||||
# The exec runner points HOME at a per-build workspace; force it to /root so abra finds its
|
||||
# server config + recipes under /root/.abra (as the manual M4/M5 runs did). Safe: capacity=1
|
||||
# means no concurrent build shares /root/.abra.
|
||||
# server config + recipes under /root/.abra (as the manual M4/M5 runs did). Safe with
|
||||
# capacity=2: app names are unique per (recipe,pr,ref) and same-recipe runs serialise on the
|
||||
# per-recipe flock, so concurrent builds never touch the same recipe checkout or app.
|
||||
HOME: /root
|
||||
commands:
|
||||
# RECIPE/REF/PR/SRC (+ CCCI_QUICK for `!testme --quick`) are injected as env vars from the
|
||||
|
||||
Reference in New Issue
Block a user