change(cleanup): reap dev deploys at start+end of /upgrade-all instead of a timer

Per operator: drop the hourly cc-ci-reap-dev-deploys systemd timer; instead run the
dev-* reaper at the START (Step 0, alongside the orphan sweep) and END (new step 4b)
of each /upgrade-all run, with THRESHOLD=0 (the run is quiescent then, so clear all
dev-* unconditionally). The reaper keeps its safe default (4h) for ad-hoc use.
Step-2b mandatory teardown is unchanged (primary mechanism); this is the backstop.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
autonomic-bot
2026-06-09 15:47:16 +00:00
co-authored by Claude Opus 4.8
parent 23bba98be4
commit c60fc6d056
4 changed files with 34 additions and 49 deletions
+14 -13
View File
@@ -2,22 +2,23 @@
# Reap LEAKED step-2b dev deploys on the cc-ci server.
#
# /recipe-upgrade step 2b deploys a recipe under a `dev-<recipe>` domain to debug an upgrade with live
# logs, and REQUIRES the agent to tear it down when done. This is the automated backstop for when that
# teardown is missed (agent crashed / killed / abandoned mid-loop): it removes `dev-*` Swarm stacks
# (+ their now-dangling volumes) whose newest service has not been updated in THRESHOLD seconds.
# logs, and REQUIRES the agent to tear it down when done. This is the backstop for a missed teardown
# (agent crashed / killed / abandoned mid-loop): it removes `dev-*` Swarm stacks (+ their dangling
# volumes). **Invoked at the START and END of an `/upgrade-all` run** (with `THRESHOLD=0` — by then the
# run is quiescent, so any `dev-*` is leftover and removed unconditionally).
#
# SAFE to run anytime — even while CI is mid-run — because it is scoped + age-gated:
# - it touches ONLY the `dev-` naming convention used by step 2b. CI per-run stacks
# (`<recipe[:4]>-<hash>`), `warm-*` canonicals, and infra are never `dev-*`, so never matched.
# - an ACTIVE dev loop redeploys (refreshing the service UpdatedAt), so it stays "fresh" and is NOT
# reaped mid-use; only an idle/abandoned `dev-*` ages past THRESHOLD and is removed.
# - volume cleanup uses `dangling=true`, so an active deploy's attached volumes are never removed.
# SAFE — scoped to the `dev-` naming convention only: CI per-run stacks (`<recipe[:4]>-<hash>`),
# `warm-*` canonicals, and infra are never `dev-*`, so never matched. Volume cleanup uses
# `dangling=true`, so a still-attached volume is never removed.
#
# Run ON the cc-ci host: ssh cc-ci 'THRESHOLD=14400 bash -s' < reap-dev-deploys.sh
# `THRESHOLD` (seconds, default 14400=4h) only removes a `dev-*` stack whose newest service has been
# idle longer than it — so an ad-hoc run while a dev loop is ACTIVE won't kill it (an active loop keeps
# redeploying, refreshing UpdatedAt). `/upgrade-all` passes `THRESHOLD=0` at run start/end to clear ALL
# leftover dev deploys. Run ON the cc-ci host: ssh cc-ci 'THRESHOLD=0 bash -s' < reap-dev-deploys.sh
set -uo pipefail
export PATH=/run/current-system/sw/bin:$PATH
THRESHOLD="${THRESHOLD:-14400}" # 4h — generous, so a long but ACTIVE dev loop is never reaped
THRESHOLD="${THRESHOLD:-14400}" # default 4h (safe for ad-hoc use); /upgrade-all passes 0 at start/end
now=$(date +%s)
reaped=0
@@ -31,8 +32,8 @@ for s in "${STACKS[@]}"; do
[ "$e" -gt "$newest" ] && newest="$e"
done
age=$(( now - newest ))
if [ "$newest" -gt 0 ] && [ "$age" -gt "$THRESHOLD" ]; then
echo "reap: dev stack '$s' idle ${age}s (> ${THRESHOLD}s) — removing"
if [ "$newest" -gt 0 ] && [ "$age" -ge "$THRESHOLD" ]; then
echo "reap: dev stack '$s' idle ${age}s (>= ${THRESHOLD}s) — removing"
docker stack rm "$s" >/dev/null 2>&1 || true
reaped=$((reaped + 1))
else