feat(cleanup): guarantee step-2b dev deploys get reaped

- /recipe-upgrade step 2b: teardown is now MANDATORY on every exit path (finally),
  with a verify-no-leak check; tear down even on failure before reporting.
- reap-dev-deploys.sh: safe, age-gated backstop that removes only idle dev-* stacks
  (never CI per-run stacks, warm-*, infra; an active dev loop stays fresh).
- orchestrator: hourly cc-ci-reap-dev-deploys systemd timer runs it against cc-ci,
  bounding any leaked dev deploy from a crashed/abandoned loop.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
autonomic-bot
2026-06-09 15:42:23 +00:00
co-authored by Claude Opus 4.8
parent 77ba7ee075
commit 23bba98be4
3 changed files with 96 additions and 3 deletions
+14 -3
View File
@@ -146,9 +146,20 @@ ssh cc-ci 'export PATH=/run/current-system/sw/bin:$PATH; set -a; . /srv/cc-ci/.t
- **Iterate:** edit the recipe → `abra app deploy $D --chaos --force` to cycle → re-read logs, until it
converges, serves, and any migration/config behaves. This is where you debug the real upgrade with
full visibility — fold what you learn back into the recipe edit (step 2).
- **Tear down when done — ALWAYS** (shared swarm): `script -qec "abra app undeploy $D -n" /dev/null`
then `script -qec "abra app rm $D -n --no-input" /dev/null` (removes volumes/secrets). The
`/upgrade-all` orphan-sweep (Step 0) is the backstop, but clean up explicitly.
- **Tear down — MANDATORY, on every exit path** (success, RED, or abort — shared swarm): the dev
deploy is yours and you MUST remove it before you finish this recipe / move to the next one. Treat
it as a `finally`:
```
ssh cc-ci 'export PATH=/run/current-system/sw/bin:$PATH; D=dev-<recipe>.ci.commoninternet.net; \
script -qec "abra app undeploy $D -n" /dev/null 2>&1 | tail -2; \
script -qec "abra app rm $D -n --no-input" /dev/null 2>&1 | tail -2'
```
Then **verify nothing leaked**: `ssh cc-ci 'docker stack ls --format "{{.Name}}" | grep -c "^dev-<recipe with . as _>"'`
should print `0` (and no `dev-<recipe>_*` volumes remain). If the recipe failed, tear the dev deploy
down anyway, THEN report the failure — never leave it running.
Backstops (defence-in-depth, NOT a substitute for the explicit teardown above): the `/upgrade-all`
orphan-sweep (Step 0) and the **hourly `cc-ci-reap-dev-deploys` timer** (reaps idle `dev-*` stacks),
so a crashed/abandoned loop's deploy is bounded — but you must still clean up yourself.
- Caveats: shared swarm — keep to **ONE** `dev-<recipe>` instance at a time and tear it down before the
next recipe; the `dev-<recipe>` domain is distinct from the harness's per-run domains and from the
`warm-*` canonicals, so the sweep removes a leaked one without touching live services.