recipe-maintainer/learnings.md

Learnings

Things discovered while working with abra and Co-op Cloud recipes in this environment.

abra TTY requirements

Several abra subcommands require a TTY and fail with "the input device is not a TTY" or "inappropriate ioctl for device" when run non-interactively. Workaround:

script -qefc "abra app backup create <domain> --chaos --no-input" /dev/null

Commands that need the TTY wrapper:

• abra app backup create
• abra app backup snapshots
• abra app restore
• abra app volume remove
• abra app cmd
• abra app logs
• abra recipe lint

Commands that work fine without a TTY:

• abra app deploy
• abra app undeploy
• abra app restart
• abra app new
• abra recipe fetch
• abra recipe versions
• abra recipe upgrade (the check mode with -m -n)

abra app ps

abra app ps fails with "inappropriate ioctl for device" when run without a TTY in its default table-output mode. Use -m (machine-readable JSON output) instead:

abra app ps <domain> --chaos --no-input -m

--chaos flag availability

Not all abra commands support --chaos. When doing local recipe development, use --chaos on every command that supports it to prevent abra from fetching remote and changing the working tree. Commands that support it:

• abra app deploy --chaos
• abra app ps --chaos
• abra app backup create --chaos
• abra app restore --chaos
• abra app restart --chaos
• abra app new --chaos
• abra app cmd --chaos
• abra app secret insert --chaos (and other secret commands)

Commands that do NOT support --chaos (and don't need it — they don't read recipe config):

• abra app undeploy
• abra app run
• abra app volume remove
• abra app backup snapshots
• abra app logs

What happens without --chaos: abra fetches the remote and checks out the latest published tag. If the working tree is dirty (unstaged/staged changes), abra refuses with FATA ... has locally unstaged changes? — so unstaged work is safe. However, if the tree is clean and you have local commits on a detached HEAD (which is how abra manages recipe checkouts), abra silently moves HEAD to the latest tag. Your commits are NOT lost — they remain in git reflog — but it looks like your work vanished. Recovery: git -C ~/.abra/recipes/<recipe> reflog to find the commit, then git checkout <hash>.

Best practice: Always use --chaos during local recipe development to avoid abra changing the working tree on you.

IMPORTANT: When abra refuses with "locally unstaged changes", do NOT commit the changes to fix it — use --chaos instead. Committing moves HEAD to a new hash, and the next non-chaos abra command will move HEAD to the latest tag, discarding your local branch position. Commits should only be made intentionally when you're ready to tag/release, not as a workaround for abra's dirty-tree check.

backup-bot-two is required for backups

abra app backup create does not perform backups itself — it delegates to a running backup-bot-two instance on the server. If backup-bot-two is not deployed, the command fails with "no backupbot discovered, is it deployed?".

Before running any backup commands, ensure backup-bot-two is deployed on the target server. It needs:

• A restic password secret
• Local storage is fine for testing (RESTIC_REPOSITORY=/backups/restic)
• Default cron schedule is 30 3 * * * (3:30 AM daily)

abra app restart requires --all-services or a service name

abra app restart <domain> alone doesn't restart anything — you must pass either --all-services / -a to restart all services, or specify a service name like app or web.

Caveat: abra app restart --all-services tends to hang on stacks with many services. Prefer abra app deploy --force to cycle services instead.

Recording and using deploy timing expectations

After a successful deploy or other long-running operation, record how long it took to converge in the recipe's test.md (e.g. "deploys converge in ~45s"). In future runs, use this as a baseline — if something takes much longer than expected, it's likely hanging and should be investigated rather than waited on indefinitely. This prevents getting stuck on operations that will never complete.

abra server setup

The server directory at ~/.abra/servers/ may exist but be empty. Running abra server add <domain> adds the server entry. The SSH config must be in ~/.ssh/config with the correct user, port, and identity file. The SSH key must be loaded into an ssh-agent for abra to use it.

Steps:

1. Copy SSH config: ensure ~/.ssh/config has the server entry
2. Set key permissions: chmod 600 <keyfile>
3. Start agent and add key: eval "$(ssh-agent -s)" && ssh-add <keyfile>
4. Add server: abra server add <domain> --no-input

abra app new

abra app new takes --server and --domain flags. The positional argument is the recipe name. User/port for SSH are read from ~/.ssh/config, not passed to app new (unlike older versions where server add accepted user and port positionally).

Recipe release notes

In Co-op Cloud, it's OK that recipe releases don't have release notes in the release/ directory if there are no breaking changes. Release notes are primarily useful for documenting breaking changes, migration steps, or significant behavioural differences that operators need to know about.

abra app run (executing commands in containers)

abra app run executes arbitrary commands inside a running container. This is different from abra app cmd, which only runs commands defined in abra.sh.

abra app run <domain> <service> -- <command> [args]

Example:

abra app run bluesky-pds.b1cc.commoninternet.net app -- goat pds admin account create --handle user.example.com

Notes:

• Does NOT support --chaos (doesn't need it — runs directly in the container)
• Use --no-tty / -t when running non-interactively (e.g. capturing output)
• Use --user / -u to run as a specific user
• Environment variables from the container (including secrets mounted at runtime) are available to the command
• Requires the TTY wrapper (script -qefc "..." /dev/null) in non-interactive environments

Docker ghost containers blocking volume removal

After undeploying a Docker Swarm stack, docker volume rm sometimes fails with "volume is in use" referencing container IDs that show as dead in docker ps -a but can't be removed with docker rm -f (returns "No such container"). This is a Docker bug — the container metadata is stale. Even docker system prune and systemctl restart docker don't always fix it.

Fix: manually remove the dead container directories and restart Docker:

ssh <server> "docker ps -a --filter volume=<volume_name> --format '{{.ID}} {{.State}}'"  # find dead containers
ssh <server> "sudo rm -rf /var/lib/docker/containers/<full_container_id>"  # remove each dead one
ssh <server> "sudo systemctl restart docker"  # restart to clear references

Prefer docker service logs over abra app logs

abra app logs tends to hang in non-interactive environments even with the TTY wrapper. Prefer using docker service logs directly via SSH instead:

ssh <server> "docker service logs <stack_name>_<service> --since 5m 2>&1"

The stack name follows the pattern <domain_with_underscores> (e.g. lasuite-docs_b1cc_commoninternet_net). The service name is appended (e.g. _minio, _backend). docker service logs also does NOT support --chaos (not an abra command), so it won't interfere with local recipe checkouts.

Golang template_driver: {{ secret "name" }} works

Docker configs with template_driver: golang support the {{ secret "name" }} function to inject Docker secrets directly into config files. This works for non-shell config formats like YAML where you can't source secrets from /run/secrets/. The secret must be assigned to the service in compose.yml. Confirmed working in the lasuite-meet recipe's livekit-server.yaml.tmpl.

Available template functions: {{ env "VAR" }} for environment variables, {{ secret "name" }} for Docker secrets.

Secrets in recipes: use abra-entrypoint.sh, not source/target mapping

Docker Swarm source/target secret mapping in compose.yml does not reliably work for renaming secrets at the mount point. Instead, use the abra-entrypoint pattern:

1. Define secrets with their short names (must pass abra lint's 12-char limit):

   secrets:
     - su_password
   

   This mounts at /run/secrets/su_password.

2. Create an abra-entrypoint.sh that reads the secret and exports it as the environment variable the application expects:

   #!/bin/sh
   set -e
   [ -f /run/secrets/su_password ] && export MUMBLE_SUPERUSER_PASSWORD="$(cat /run/secrets/su_password)"
   exec /entrypoint.sh "$@"
   

3. Mount it as a Docker config and override the entrypoint:

   entrypoint: ["/abra-entrypoint.sh"]
   command: ["/usr/bin/mumble-server"]  # the original CMD
   configs:
     - source: abra_entrypoint
       target: /abra-entrypoint.sh
       mode: 0555
   

4. The abra-entrypoint.sh must chain into the original image entrypoint (check the upstream Dockerfile's ENTRYPOINT) via exec /entrypoint.sh "$@".

5. Config versions (e.g. ABRA_ENTRYPOINT_VERSION) go in abra.sh as export statements. Secret versions (e.g. SECRET_SU_PASSWORD_VERSION) go in .env.sample.

See lasuite-meet/abra-entrypoint.sh for the canonical example.

WebFetch and JavaScript-heavy apps

WebFetch does not execute JavaScript. Apps like CryptPad that are fully client-side encrypted will return a "JavaScript must be enabled" page. This is expected and healthy — a curl HTTP 200 check is sufficient to verify the app is running. Don't treat the JavaScript-required message as a failure.

Never skip OIDC integration tests

When testing a recipe — whether it's a migration test, upgrade test, or any other test context — always run the full OIDC integration tests if they exist. The OIDC integration tests are part of the recipe's test suite and must be set up and executed, not skipped. Set up the OIDC provider/integration before starting the test sequence so it can be verified at every stage.