autonomic-bot
f283a371bb
Sanitized single-commit public mirror of recipe-maintainer. - Removed test-ssh/.testenv (live creds); added test-ssh/.testenv.example placeholders. - Removed plans/ and planned-updates/ (deployment-planning docs) so no client/ deployment domains appear in the public repo. - All other secret stores were already gitignored. - docs.coopcloud.tech retained as a submodule (public upstream).
111 lines
3.6 KiB
Markdown
111 lines
3.6 KiB
Markdown
---
|
|
description: Run tests for all maintained recipes, deploying each one at a time
|
|
allowed-tools: [Bash, Read, Write, Edit, Glob, Grep, WebFetch]
|
|
---
|
|
|
|
# Recipe Test All
|
|
|
|
Deploy and test every maintained recipe on the active test instance, one at a time. For each recipe, free server memory by undeploying unrelated apps, deploy the recipe and its dependencies, run all its tests, then move on to the next.
|
|
|
|
Read and follow the instructions in `.claude/commands/includes/logging.md`.
|
|
Read and follow the instructions in `.claude/commands/includes/guidelines.md`.
|
|
|
|
## Steps
|
|
|
|
### 1. Determine the active instance
|
|
|
|
Run `python3 scripts/get_test_instance.py` to get SERVER and INSTANCE. Read `settings.toml` to get `domain_suffix`.
|
|
|
|
### 2. Discover all recipes and build deployment tiers
|
|
|
|
Glob `recipe-info/*/recipe.toml`. For each, read the file to get:
|
|
- `name` — the recipe name
|
|
- `[dependencies].requires` — list of recipe dependencies (may be empty or absent)
|
|
|
|
Group into tiers:
|
|
- **Tier 1** (no dependencies): recipes with empty or missing `requires`
|
|
- **Tier 2** (depends on tier 1): recipes whose `requires` list only contains tier 1 recipes
|
|
|
|
Present the test order to the user and **confirm before proceeding**.
|
|
|
|
### 3. Test each recipe in dependency order
|
|
|
|
Process recipes **one at a time** within each tier. For each recipe:
|
|
|
|
#### a. Context reset
|
|
|
|
Free memory by undeploying everything except infrastructure and this recipe's dependencies:
|
|
```bash
|
|
python3 scripts/context_reset.py --recipe <recipe>
|
|
```
|
|
|
|
#### b. Deploy dependencies and test dependencies
|
|
|
|
If the recipe has `[dependencies].requires` (runtime deps), deploy each one:
|
|
```bash
|
|
abra app deploy <dep-domain> --chaos --force --no-input
|
|
```
|
|
|
|
If the recipe has `[dependencies].test_requires` (test-only deps), deploy those too. These are domain prefixes (e.g. `lasuite-docs`, `ld2`) — the full domain is `<prefix>.<DOMAIN_SUFFIX>`. Note: `test_requires` are NOT transitive — only the target recipe's test_requires are deployed, not those of its dependencies.
|
|
|
|
Wait for each dependency to become healthy before continuing.
|
|
|
|
#### c. Deploy this recipe
|
|
|
|
```bash
|
|
abra app deploy <recipe>.<DOMAIN_SUFFIX> --chaos --force --no-input
|
|
```
|
|
|
|
If the deploy reports failure but services are running (check via SSH `docker service ls`), treat it as a success — abra's convergence checker times out on complex multi-service stacks.
|
|
|
|
If the deploy truly fails (services not starting, missing secrets, etc.), log FAIL for this recipe and skip to the next one.
|
|
|
|
#### d. Run tests
|
|
|
|
Discover all test scripts:
|
|
```bash
|
|
ls recipe-info/<recipe>/tests/*.py
|
|
```
|
|
|
|
Run each test script in sequence:
|
|
```bash
|
|
python3 recipe-info/<recipe>/tests/<script>.py
|
|
```
|
|
|
|
Capture stdout/stderr and exit code for each script.
|
|
|
|
#### e. Log results
|
|
|
|
For each test script, log:
|
|
- Script name
|
|
- PASS (exit 0) or FAIL (non-zero exit)
|
|
- Key output lines (especially PASS:/FAIL: messages)
|
|
|
|
### 4. Summary report
|
|
|
|
After all recipes are tested, print a summary table:
|
|
|
|
```
|
|
## Test Results — <INSTANCE>.commoninternet.net — <DATE>
|
|
|
|
| Recipe | Tier | Deploy | Tests | Details |
|
|
|--------|------|--------|-------|---------|
|
|
| keycloak | 1 | PASS | 2/2 | health_check PASS, oidc_integration PASS |
|
|
| authentik | 1 | PASS | 1/2 | health_check PASS, oidc_integration FAIL |
|
|
| ... | ... | ... | ... | ... |
|
|
|
|
**Overall: X/Y recipes fully passing**
|
|
```
|
|
|
|
Where:
|
|
- **Deploy**: PASS if the app deployed and responded to health check, FAIL otherwise
|
|
- **Tests**: `passed/total` count of test scripts
|
|
- **Details**: one-line summary of each test script result
|
|
|
|
### 5. Write the log
|
|
|
|
Save the full log (including all commands, outputs, and the summary table) to:
|
|
```
|
|
logs/recipe-test-all-<YYYY-MM-DD>.md
|
|
```
|