---
description: Run tests for all maintained recipes, deploying each one at a time
allowed-tools: [Bash, Read, Write, Edit, Glob, Grep, WebFetch]
---

# Recipe Test All

Deploy and test every maintained recipe on the active test instance, one at a time. For each recipe, free server memory by undeploying unrelated apps, deploy the recipe and its dependencies, run all its tests, then move on to the next.

Read and follow the instructions in `.claude/commands/includes/logging.md`.
Read and follow the instructions in `.claude/commands/includes/guidelines.md`.

## Steps

### 1. Determine the active instance

Run `python3 scripts/get_test_instance.py` to get SERVER and INSTANCE. Read `settings.toml` to get `domain_suffix`.

### 2. Discover all recipes and build deployment tiers

Glob `recipe-info/*/recipe.toml`. For each, read the file to get:
- `name` — the recipe name
- `[dependencies].requires` — list of recipe dependencies (may be empty or absent)

Group into tiers:
- **Tier 1** (no dependencies): recipes with empty or missing `requires`
- **Tier 2** (depends on tier 1): recipes whose `requires` list only contains tier 1 recipes

Present the test order to the user and **confirm before proceeding**.

### 3. Test each recipe in dependency order

Process recipes **one at a time** within each tier. For each recipe:

#### a. Context reset

Free memory by undeploying everything except infrastructure and this recipe's dependencies:
```bash
python3 scripts/context_reset.py --recipe <recipe>
```

#### b. Deploy dependencies and test dependencies

If the recipe has `[dependencies].requires` (runtime deps), deploy each one:
```bash
abra app deploy <dep-domain> --chaos --force --no-input
```

If the recipe has `[dependencies].test_requires` (test-only deps), deploy those too. These are domain prefixes (e.g. `lasuite-docs`, `ld2`) — the full domain is `<prefix>.<DOMAIN_SUFFIX>`. Note: `test_requires` are NOT transitive — only the target recipe's test_requires are deployed, not those of its dependencies.

Wait for each dependency to become healthy before continuing.

#### c. Deploy this recipe

```bash
abra app deploy <recipe>.<DOMAIN_SUFFIX> --chaos --force --no-input
```

If the deploy reports failure but services are running (check via SSH `docker service ls`), treat it as a success — abra's convergence checker times out on complex multi-service stacks.

If the deploy truly fails (services not starting, missing secrets, etc.), log FAIL for this recipe and skip to the next one.

#### d. Run tests

Discover all test scripts:
```bash
ls recipe-info/<recipe>/tests/*.py
```

Run each test script in sequence:
```bash
python3 recipe-info/<recipe>/tests/<script>.py
```

Capture stdout/stderr and exit code for each script.

#### e. Log results

For each test script, log:
- Script name
- PASS (exit 0) or FAIL (non-zero exit)
- Key output lines (especially PASS:/FAIL: messages)

### 4. Summary report

After all recipes are tested, print a summary table:

```
## Test Results — <INSTANCE>.commoninternet.net — <DATE>

| Recipe | Tier | Deploy | Tests | Details |
|--------|------|--------|-------|---------|
| keycloak | 1 | PASS | 2/2 | health_check PASS, oidc_integration PASS |
| authentik | 1 | PASS | 1/2 | health_check PASS, oidc_integration FAIL |
| ... | ... | ... | ... | ... |

**Overall: X/Y recipes fully passing**
```

Where:
- **Deploy**: PASS if the app deployed and responded to health check, FAIL otherwise
- **Tests**: `passed/total` count of test scripts
- **Details**: one-line summary of each test script result

### 5. Write the log

Save the full log (including all commands, outputs, and the summary table) to:
```
logs/recipe-test-all-<YYYY-MM-DD>.md
```