feat(cleanup): guarantee step-2b dev deploys get reaped
- /recipe-upgrade step 2b: teardown is now MANDATORY on every exit path (finally), with a verify-no-leak check; tear down even on failure before reporting. - reap-dev-deploys.sh: safe, age-gated backstop that removes only idle dev-* stacks (never CI per-run stacks, warm-*, infra; an active dev loop stays fresh). - orchestrator: hourly cc-ci-reap-dev-deploys systemd timer runs it against cc-ci, bounding any leaked dev deploy from a crashed/abandoned loop. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Opus 4.8
parent
77ba7ee075
commit
23bba98be4
@@ -146,9 +146,20 @@ ssh cc-ci 'export PATH=/run/current-system/sw/bin:$PATH; set -a; . /srv/cc-ci/.t
|
||||
- **Iterate:** edit the recipe → `abra app deploy $D --chaos --force` to cycle → re-read logs, until it
|
||||
converges, serves, and any migration/config behaves. This is where you debug the real upgrade with
|
||||
full visibility — fold what you learn back into the recipe edit (step 2).
|
||||
- **Tear down when done — ALWAYS** (shared swarm): `script -qec "abra app undeploy $D -n" /dev/null`
|
||||
then `script -qec "abra app rm $D -n --no-input" /dev/null` (removes volumes/secrets). The
|
||||
`/upgrade-all` orphan-sweep (Step 0) is the backstop, but clean up explicitly.
|
||||
- **Tear down — MANDATORY, on every exit path** (success, RED, or abort — shared swarm): the dev
|
||||
deploy is yours and you MUST remove it before you finish this recipe / move to the next one. Treat
|
||||
it as a `finally`:
|
||||
```
|
||||
ssh cc-ci 'export PATH=/run/current-system/sw/bin:$PATH; D=dev-<recipe>.ci.commoninternet.net; \
|
||||
script -qec "abra app undeploy $D -n" /dev/null 2>&1 | tail -2; \
|
||||
script -qec "abra app rm $D -n --no-input" /dev/null 2>&1 | tail -2'
|
||||
```
|
||||
Then **verify nothing leaked**: `ssh cc-ci 'docker stack ls --format "{{.Name}}" | grep -c "^dev-<recipe with . as _>"'`
|
||||
should print `0` (and no `dev-<recipe>_*` volumes remain). If the recipe failed, tear the dev deploy
|
||||
down anyway, THEN report the failure — never leave it running.
|
||||
Backstops (defence-in-depth, NOT a substitute for the explicit teardown above): the `/upgrade-all`
|
||||
orphan-sweep (Step 0) and the **hourly `cc-ci-reap-dev-deploys` timer** (reaps idle `dev-*` stacks),
|
||||
so a crashed/abandoned loop's deploy is bounded — but you must still clean up yourself.
|
||||
- Caveats: shared swarm — keep to **ONE** `dev-<recipe>` instance at a time and tear it down before the
|
||||
next recipe; the `dev-<recipe>` domain is distinct from the harness's per-run domains and from the
|
||||
`warm-*` canonicals, so the sweep removes a leaked one without touching live services.
|
||||
|
||||
Reference in New Issue
Block a user