--- description: Test backing up and restoring a recipe's test instance argument-hint: allowed-tools: [Bash, Read, Write, Glob, Grep, WebFetch] --- # Recipe Backup Test Test the full backup and restore cycle for a Co-op Cloud recipe: create a backup, tear down the test instance (undeploy + remove volumes), redeploy from scratch, restore from the backup, and verify the app still works. **Important:** All `abra` commands that read the recipe (deploy, backup, restore, restart, ps) MUST use `--chaos` so they use the current local recipe checkout, including any uncommitted changes. This ensures we're testing the backup/restore labels as they exist in the working copy. **TTY workaround:** Several `abra` subcommands (`backup create`, `backup snapshots`, `restore`, `volume remove`) fail with "the input device is not a TTY" in non-interactive environments. Wrap these with `script -qefc "..." /dev/null` to provide a pseudo-TTY. The recipe name is: $ARGUMENTS Read and follow the instructions in `.claude/commands/includes/logging.md`. Read and follow the instructions in `.claude/commands/includes/guidelines.md`. Read `~/.abra/recipes/backup-bot-two/README.md` for context on how backup-bot-two works (its CLI commands, backup/restore flow, and label conventions). ## Steps 1. **Free server resources** by running `/test-context-reset $ARGUMENTS` to undeploy unrelated apps from the test server while keeping this recipe and its dependencies running. 2. **Get the domain and server for this recipe**: ``` python3 scripts/get_test_instance.py --recipe $ARGUMENTS ``` This outputs DOMAIN and SERVER for the active instance. Also read `settings.toml` to get `DOMAIN_SUFFIX` (the `domain_suffix` field for the active instance). - If the recipe has no `recipe-info/$ARGUMENTS/recipe.toml`, tell the user to run `/recipe-init $ARGUMENTS` first and stop. 3. **Read recipe-specific backup test notes** from `recipe-info/$ARGUMENTS/tests/backup-test.md`, if it exists. - This optional file can contain recipe-specific instructions: extra data-seeding steps before the backup, services to check after restore, known caveats, additional verification commands, or anything else specific to testing this recipe's backup/restore cycle. - Keep these notes in mind and apply them at the relevant steps below. 4. **Fetch the recipe and check for backup labels**: - Fetch the recipe (check for local changes first — see guidelines). - Read `~/.abra/recipes/$ARGUMENTS/compose.yml`. - Search for `backupbot.backup` deploy labels. - If **no backup labels** are found anywhere in the compose file: - Tell the user this recipe does not have backup configured. - Explain what's needed: at minimum, a `backupbot.backup=true` deploy label on the main service, plus appropriate pre/post hooks for databases (see the Co-op Cloud backup spec in `docs.coopcloud.tech/docs/specs/backup/`). - Stop here. - If backup labels are found, **summarise the backup configuration**: - Which service has `backupbot.backup=true` - Any `backupbot.backup.pre-hook` / `backupbot.backup.post-hook` commands - Any `backupbot.restore.pre-hook` / `backupbot.restore.post-hook` commands - Any volume/path restrictions (`backupbot.backup.path`, `backupbot.backup.volumes.*`) 5. **Ensure backup-bot-two is deployed on the server**: - `abra app backup create` requires backup-bot-two to be running on the server. Check by running: ``` abra app ls --server --status 2>&1 | grep backup-bot ``` - If backup-bot-two is not deployed, deploy it: - `abra recipe fetch backup-bot-two --force` - `abra app new backup-bot-two --server --domain backupbot. --secrets --no-input` - `abra app deploy backupbot. --force --no-input` - If `app new` says it already exists, just check it's deployed and deploy if not. - Confirm backup-bot-two is running before proceeding. 6. **Verify the app is deployed and healthy**: - Run `abra app ps --chaos --no-input -m` to check deployment status (use `-m` for machine-readable output to avoid TTY issues). - Run the existing health check script if available: `python3 recipe-info/$ARGUMENTS/tests/health_check.py`. - If no health check script exists, curl `https://` and check for HTTP 200. - If the app is not deployed, deploy it first: `abra app deploy --chaos --force --no-input`, then wait and re-check. - If the backup labels were just added and the app was already deployed, **redeploy** so the running services pick up the new labels: `abra app deploy --chaos --force --no-input`. - If `backup-test.md` specifies any data-seeding steps to perform before the backup (e.g. creating a test document, inserting a database record), do them now. 7. **Create a backup snapshot**: - First, list existing snapshots to establish a baseline: `script -qefc "abra app backup snapshots --no-input" /dev/null` - Note: `backup snapshots` does NOT support `--chaos`. - Note the number of existing snapshots (may be zero). - Run (with TTY wrapper): `script -qefc "abra app backup create --chaos --no-input" /dev/null` - Confirm the output contains "backup finished" or similar success message. - List snapshots again: `script -qefc "abra app backup snapshots --no-input" /dev/null` - Confirm there is exactly one more snapshot than before. - Note the newest snapshot ID from the output. 8. **Verify backup contents**: - Run `backup ls` inside the backupbot container to list the files in the latest snapshot, filtered to this app: ``` script -qefc "abra app run backupbot. app -- backup -h ls" /dev/null ``` - The `-h ` flag filters the listing to only this app's snapshot. The `ls` command defaults to listing under `/var/lib/docker/volumes/`. - Note: `abra app run` requires the TTY wrapper and does NOT support `--chaos`. - Save the output, then use `grep` to confirm that expected files/paths are present: - **Volume paths**: For each volume identified as backed up in step 3, grep the output for the volume name (the Docker volume name is `_`). Confirm that at least one file appears under each expected volume path. - **Excluded volumes**: For any volumes explicitly excluded (e.g. `backupbot.backup.volumes.=false`), confirm they do NOT appear in the listing. - **Recipe-specific files**: If `backup-test.md` specifies particular files or patterns to look for in the backup, grep for those as well. - If any expected volume paths are missing or any excluded volumes appear, flag as **FAIL** with details about what was missing or unexpectedly present. 9. **Undeploy the app**: - Run: `abra app undeploy --no-input` - Undeploy does not need `--chaos`. 10. **Remove volumes** to simulate complete data loss: - Run (with TTY wrapper): `script -qefc "abra app volume remove --force --no-input" /dev/null` - This ensures the restore test starts from a genuinely clean slate — no leftover data. - **Do not skip this step** — testing restore onto existing data is not a valid backup test. - If volume removal fails with "volume is in use" by dead/ghost containers that Docker can't remove, clear them manually via SSH: 1. Identify the ghost containers: `ssh "docker ps -a --filter volume= --format '{{.ID}} {{.State}}'"` — they will show as `dead` 2. Remove their directories: `ssh "sudo rm -rf /var/lib/docker/containers/"` 3. Restart Docker: `ssh "sudo systemctl restart docker"` 4. Wait ~15 seconds, then retry volume removal 11. **Redeploy from scratch**: - Run: `abra app deploy --chaos --force --no-input` - Wait for the app to come up (check with `abra app ps --chaos --no-input -m`). - The app may not be fully healthy yet since it has no data — that's expected. 12. **Restore from backup**: - Run (with TTY wrapper): `script -qefc "abra app restore --chaos --hooks --no-input" /dev/null` - The `--hooks` flag ensures restore pre/post hooks run (e.g. database import commands). - **Do NOT pass `--target`** to the restore command. The `--target` flag is a restic restore target *directory*, not a stack filter — passing a stack name as `--target` causes restic to restore files into a wrong subdirectory instead of the actual volume paths, and restore hooks will fail because the files won't be where the containers expect them. - Confirm the output contains "Restoring Snapshot" or similar success message. 13. **Redeploy the app** to ensure it picks up the restored data: - Do NOT use `abra app restart --all-services` — it tends to hang on stacks with many services. - Instead, redeploy with force: `abra app deploy --chaos --force --no-input` - Wait for all services to converge: `sleep 30`, then check with `abra app ps --chaos --no-input -m`. Allow up to 60 seconds for services to start. If abra reports a deploy timeout but `app ps` shows all services running and healthy, treat it as a pass. 14. **Run the test suite** to verify the app works after restore: - Discover and run all test scripts from `recipe-info/$ARGUMENTS/tests/*.py`. - For each script, record PASS (exit 0) or FAIL (non-zero). - Read `recipe-info/$ARGUMENTS/test.md` and perform URL-based checks using `curl` or `WebFetch`. - If `backup-test.md` specifies extra verification steps (e.g. "confirm the test document still exists", "check the database has records"), perform those too. - If no tests exist at all, at minimum curl `https://` and check for HTTP 200. 15. **Summarise results**: Report each phase of the backup/restore cycle: | Phase | Result | |-------|--------| | Backup-bot-two check | PASS / FAIL | | Initial health check | PASS / FAIL | | Backup creation | PASS / FAIL | | Backup contents verification | PASS / FAIL | | Undeploy + volume removal | PASS / FAIL | | Fresh redeploy | PASS / FAIL | | Restore from backup | PASS / FAIL | | Post-restore redeploy | PASS / FAIL | | Test suite | PASS / FAIL (detail per test) | - If all phases passed: confirm the recipe's backup/restore cycle is working correctly. - If any phase failed: highlight which step failed, show relevant error output, and suggest troubleshooting: - Check backup labels in compose.yml - Check pre/post hook commands for errors - Review app logs: `abra app logs ` - Check if all volumes are being backed up - For database services, verify the dump/restore hooks are correct