recipe-maintainer/.claude/commands/recipe-test-all.md

description, allowed-tools

description	allowed-tools
Run tests for all maintained recipes, deploying each one at a time	Bash Read Write Edit Glob Grep WebFetch

Recipe Test All

Deploy and test every maintained recipe on the active test instance, one at a time. For each recipe, free server memory by undeploying unrelated apps, deploy the recipe and its dependencies, run all its tests, then move on to the next.

Read and follow the instructions in .claude/commands/includes/logging.md. Read and follow the instructions in .claude/commands/includes/guidelines.md.

Steps

1. Determine the active instance

Run python3 scripts/get_test_instance.py to get SERVER and INSTANCE. Read settings.toml to get domain_suffix.

2. Discover all recipes and build deployment tiers

Glob recipe-info/*/recipe.toml. For each, read the file to get:

• name — the recipe name
• [dependencies].requires — list of recipe dependencies (may be empty or absent)

Group into tiers:

• Tier 1 (no dependencies): recipes with empty or missing requires
• Tier 2 (depends on tier 1): recipes whose requires list only contains tier 1 recipes

Present the test order to the user and confirm before proceeding.

3. Test each recipe in dependency order

Process recipes one at a time within each tier. For each recipe:

a. Context reset

Free memory by undeploying everything except infrastructure and this recipe's dependencies:

python3 scripts/context_reset.py --recipe <recipe>

b. Deploy dependencies and test dependencies

If the recipe has [dependencies].requires (runtime deps), deploy each one:

abra app deploy <dep-domain> --chaos --force --no-input

If the recipe has [dependencies].test_requires (test-only deps), deploy those too. These are domain prefixes (e.g. lasuite-docs, ld2) — the full domain is <prefix>.<DOMAIN_SUFFIX>. Note: test_requires are NOT transitive — only the target recipe's test_requires are deployed, not those of its dependencies.

Wait for each dependency to become healthy before continuing.

c. Deploy this recipe

abra app deploy <recipe>.<DOMAIN_SUFFIX> --chaos --force --no-input

If the deploy reports failure but services are running (check via SSH docker service ls), treat it as a success — abra's convergence checker times out on complex multi-service stacks.

If the deploy truly fails (services not starting, missing secrets, etc.), log FAIL for this recipe and skip to the next one.

d. Run tests

Discover all test scripts:

ls recipe-info/<recipe>/tests/*.py

Run each test script in sequence:

python3 recipe-info/<recipe>/tests/<script>.py

Capture stdout/stderr and exit code for each script.

e. Log results

For each test script, log:

• Script name
• PASS (exit 0) or FAIL (non-zero exit)
• Key output lines (especially PASS:/FAIL: messages)

4. Summary report

After all recipes are tested, print a summary table:

## Test Results — <INSTANCE>.commoninternet.net — <DATE>

| Recipe | Tier | Deploy | Tests | Details |
|--------|------|--------|-------|---------|
| keycloak | 1 | PASS | 2/2 | health_check PASS, oidc_integration PASS |
| authentik | 1 | PASS | 1/2 | health_check PASS, oidc_integration FAIL |
| ... | ... | ... | ... | ... |

**Overall: X/Y recipes fully passing**

Where:

• Deploy: PASS if the app deployed and responded to health check, FAIL otherwise
• Tests: passed/total count of test scripts
• Details: one-line summary of each test script result

5. Write the log

Save the full log (including all commands, outputs, and the summary table) to:

logs/recipe-test-all-<YYYY-MM-DD>.md