feat(discourse): switch app to official discourse/discourse image (experimental)

Replaces the paywalled bitnamilegacy app with the official discourse/discourse
image behind Traefik. DB is reused as-is; uploads migrate from the legacy
bitnami volume idempotently. The wrapper entrypoint injects the db_password and
smtp_password secrets (the official image has no *_FILE support). SMTP env vars
are renamed to the official names; release notes cover the migration.

Recipe 0.8.1+3.5.0 -> 1.0.0+3.5.3 (major: new image, env/volume/port changes).

.drone.yml

@@ -3,7 +3,7 @@ kind: pipeline
 name: deploy to swarm-test.autonomic.zone
 steps:
   - name: deployment
-    image: decentral1se/stack-ssh-deploy:latest
+    image: git.coopcloud.tech/coop-cloud/stack-ssh-deploy:latest
     settings:
       host: swarm-test.autonomic.zone
       stack: discourse
@@ -11,11 +11,30 @@ steps:
       purge: true
       deploy_key:
         from_secret: drone_ssh_swarm_test
+      networks:
+        - proxy
     environment:
       DOMAIN: discourse.swarm-test.autonomic.zone
       STACK_NAME: discourse
       LETS_ENCRYPT_ENV: production
       SECRET_DB_PASSWORD_VERSION: v1
+      DB_ENTRYPOINT_VERSION: v1
 trigger:
   branch:
     - main
+---
+kind: pipeline
+name: generate recipe catalogue
+steps:
+  - name: release a new version
+    image: plugins/downstream
+    settings:
+      server: https://build.coopcloud.tech
+      token:
+        from_secret: drone_abra-bot_token
+      fork: true
+      repositories:
+        - toolshed/auto-recipes-catalogue-json
+
+trigger:
+  event: tag

.env.sample

@@ -5,15 +5,19 @@ DOMAIN=discourse.example.com
 #EXTRA_DOMAINS=', `www.discourse.example.com`'
 LETS_ENCRYPT_ENV=production
 
-# Outgoing email
+# Admin / developer accounts (comma-separated); these become admins on signup
-#DISCOURSE_SMTP_HOST=
+DISCOURSE_DEVELOPER_EMAILS=admin@example.com
-#DISCOURSE_SMTP_PORT=
-#DISCOURSE_SMTP_USER=
-#DISCOURSE_SMTP_PROTOCOL=
-#DISCOURSE_SMTP_AUTH=
 
-# SMTP authentication
+# Outgoing email (official discourse/discourse env names)
-#COMPOSE_FILE="compose.yml:compose.smtpauth.yml"
+#DISCOURSE_SMTP_ADDRESS=
+#DISCOURSE_SMTP_PORT=587
+#DISCOURSE_SMTP_USER_NAME=
+#DISCOURSE_SMTP_AUTHENTICATION=login
+#DISCOURSE_SMTP_ENABLE_START_TLS=true
+# Set this if you send e-mail from a different address than noreply@$DOMAIN
+#DISCOURSE_NOTIFICATION_EMAIL=
+
+# SMTP password as a secret
 #SECRET_SMTP_PASSWORD_VERSION=v1
 
 SECRET_DB_PASSWORD_VERSION=v1

README.md

@@ -6,42 +6,54 @@ A platform for community discussion
 
 <!-- metadata -->
 * **Category**: Apps
-* **Status**: 
+* **Status**: 3, experimental
-* **Image**: [`bitnami/discourse`](https://hub.docker.com/r/bitname/discourse)
+* **Image**: [`discourse/discourse`](https://hub.docker.com/r/discourse/discourse), 4, upstream
 * **Healthcheck**: yes
-* **Backups**: no
+* **Backups**: yes
 * **Email**: yes
-* **Tests**: no
+* **Tests**: yes
 * **SSO**: no
 <!-- endmetadata -->
 
+> **Note**: this recipe runs the official, **experimental** `discourse/discourse`
+> image. Upstream does not yet recommend it for production — see
+> <https://meta.discourse.org/t/380646>. Use with care.
+
 ## Basic usage
 
 1. Set up Docker Swarm and [`abra`]
 2. Deploy [`coop-cloud/traefik`]
-3. `abra app new discourse --secrets` (optionally with `--pass` if you'd like
+3. `abra app new discourse --secrets`
-   to save secrets in `pass`)
+4. `abra app config YOURAPPDOMAIN` — set `DOMAIN` and `DISCOURSE_DEVELOPER_EMAILS`
-4. `abra app YOURAPPDOMAIN config` - be sure to change `$DOMAIN` to something that resolves to
+5. `abra app deploy YOURAPPDOMAIN`
-   your Docker swarm box
+6. Open the configured domain in your browser to finish set-up. The first account
-5. `abra app YOURAPPDOMAIN deploy`
+   that registers with an address listed in `DISCOURSE_DEVELOPER_EMAILS` becomes
-6. Open the configured domain in your browser to finish set-up
+   an admin.
 
-[`abra`]: https://git.autonomic.zone/autonomic-cooperative/abra
+[`abra`]: https://docs.coopcloud.tech/abra/
-[`coop-cloud/traefik`]: https://git.autonomic.zone/coop-cloud/traefik
+[`coop-cloud/traefik`]: https://git.coopcloud.tech/coop-cloud/traefik
 
-## To add a new Admin user
+The app serves plain HTTP on port 80; Traefik terminates TLS in front of it. The
+image's built-in nginx/Let's Encrypt is disabled by the recipe (`install-ssl`
+override) so it works behind the reverse proxy.
 
-1. Login to the instance `abra app run APPNAME app sh`
+## Add an admin user
-2. `cd /opt/bitnami/discourse`
-3. `RAILS_ENV=production bundle exec rake admin:create` and follow prompts.
 
-## Install plugins
+```
+abra app run YOURAPPDOMAIN app discourse admin create
+```
 
-1. Login to instance `abra app run APPNAME app sh`
+## Migrating from the previous (bitnami) recipe
-2. `cd /bitnami/discourse/plugins/`
-3. `git clone plugingit` for example `https://github.com/discourse/discourse-openid-connect.git`
-4. `abra app restart APPNAME app`
 
-## Setup Notes
+The official image stores uploads under `/shared` rather than bitnami's
+`/bitnami/discourse`. On first boot the recipe copies uploads + backups from the
+old bitnami volume (mounted read-only at `/legacy`) into `/shared`, once,
+idempotently. The Postgres database is reused as-is. After a successful migration
+a later recipe version will drop the transitional `/legacy` mount.
 
-Until issue #1 is fixed, the default user is `user` and the default password is `bitnami123`
+If you are upgrading from the bitnami recipe, also remove the now-unused sidekiq
+service that swarm leaves behind (sidekiq runs inside the app container now):
+
+```
+docker service rm YOURSTACK_sidekiq
+```

abra.sh

@@ -0,0 +1,5 @@
+export DB_ENTRYPOINT_VERSION=v3
+export PG_BACKUP_VERSION=v2
+export APP_ENTRYPOINT_VERSION=v2
+export APP_INSTALL_SSL_VERSION=v1
+export APP_MIGRATE_UPLOADS_VERSION=v1

app-install-ssl.sh

@@ -0,0 +1,11 @@
+#!/bin/bash
+# Overrides the official image's /etc/runit/1.d/install-ssl.
+#
+# The stock install-ssl always runs configure-ssl (and configure-letsencrypt),
+# which empties the default `listen 80` nginx outlet and switches to `listen 443
+# ssl` against a cert that does not exist here — nginx then crash-loops, or the
+# image tries to obtain its own Let's Encrypt cert. Under Co-op Cloud, Traefik
+# terminates TLS and proxies plain HTTP to port 80, so we skip the image's SSL
+# setup entirely and let nginx keep its default HTTP-on-80 config.
+echo "install-ssl overridden by recipe: serving plain HTTP on :80 behind Traefik"
+exit 0

cc-app-entrypoint.sh

@@ -0,0 +1,15 @@
+#!/bin/bash
+# Co-op Cloud wrapper around the official image's /sbin/boot.
+# discourse/discourse reads passwords from the process env (pups/Ruby; it has no
+# *_FILE support), so inject them from the docker secrets before booting.
+set -e
+
+if [ -f /run/secrets/db_password ]; then
+  export DISCOURSE_DB_PASSWORD="$(cat /run/secrets/db_password)"
+fi
+
+if [ -f /run/secrets/smtp_password ]; then
+  export DISCOURSE_SMTP_PASSWORD="$(cat /run/secrets/smtp_password)"
+fi
+
+exec /sbin/boot

compose.smtpauth.yml

@@ -3,14 +3,8 @@ version: "3.8"
 
 services:
   app:
-    environment:
+    # the wrapper entrypoint reads /run/secrets/smtp_password and exports
-      - DISCOURSE_SMTP_PASSWORD_FILE=/var/run/secrets/smtp_password
+    # DISCOURSE_SMTP_PASSWORD (the official image has no *_FILE support)
-    secrets:
-      - smtp_password
-
-  sidekiq:
-    environment:
-      - DISCOURSE_SMTP_PASSWORD_FILE=/var/run/secrets/smtp_password
     secrets:
       - smtp_password
 

compose.yml

@@ -3,98 +3,114 @@ version: "3.8"
 
 services:
   app:
-    image: bitnami/discourse:2.7.7
+    image: discourse/discourse:3.5.3
     networks:
       - proxy
       - internal
-    # entrypoint: ['tail', '-f', '/dev/null']
+    # official image CMD is /sbin/boot; wrapper injects the DB password secret first
+    entrypoint: /usr/local/bin/cc-app-entrypoint.sh
     environment:
-      - ALLOW_EMPTY_PASSWORD=yes
+      - DISCOURSE_HOSTNAME=${DOMAIN}
-      - DISCOURSE_HOST=${DOMAIN}
+      - DISCOURSE_DEVELOPER_EMAILS=${DISCOURSE_DEVELOPER_EMAILS}
-      - DISCOURSE_DATABASE_HOST=db
+      - DISCOURSE_DB_HOST=${STACK_NAME}_db
-      - DISCOURSE_DATABASE_USER=discourse
+      - DISCOURSE_DB_PORT=5432
-      - DISCOURSE_DATABASE_NAME=discourse
+      - DISCOURSE_DB_NAME=discourse
-      - DISCOURSE_DATABASE_PASSWORD_FILE=/run/secrets/db_password
+      - DISCOURSE_DB_USERNAME=discourse
-      - DISCOURSE_SMTP_HOST
+      - DISCOURSE_REDIS_HOST=${STACK_NAME}_redis
+      - DISCOURSE_REDIS_PORT=6379
+      - DISCOURSE_SMTP_ADDRESS
       - DISCOURSE_SMTP_PORT
-      - DISCOURSE_SMTP_USER
+      - DISCOURSE_SMTP_USER_NAME
-      - DISCOURSE_SMTP_PROTOCOL
+      - DISCOURSE_SMTP_PASSWORD
-      - DISCOURSE_SMTP_AUTH
+      - DISCOURSE_SMTP_AUTHENTICATION
+      - DISCOURSE_SMTP_ENABLE_START_TLS
+      - DISCOURSE_NOTIFICATION_EMAIL
     volumes:
-      - 'discourse_data:/bitnami/discourse'
+      - 'discourse_shared:/shared'
+      # transition only: legacy bitnami volume, read-only, for one-time upload migration
+      - 'discourse_data:/legacy:ro'
     secrets:
       - db_password
+    configs:
+      - source: app_entrypoint
+        target: /usr/local/bin/cc-app-entrypoint.sh
+        mode: 0555
+      - source: app_install_ssl
+        target: /etc/runit/1.d/install-ssl
+        mode: 0555
+      - source: app_migrate_uploads
+        target: /etc/runit/1.d/02-migrate-bitnami-uploads
+        mode: 0555
     depends_on:
       - db
       - redis
     deploy:
       update_config:
         failure_action: rollback
-        order: start-first
+        order: stop-first
       labels:
         - "traefik.enable=true"
-        - "traefik.http.services.${STACK_NAME}.loadbalancer.server.port=3000"
+        - "traefik.http.services.${STACK_NAME}.loadbalancer.server.port=80"
         - "traefik.http.routers.${STACK_NAME}.rule=Host(`${DOMAIN}`${EXTRA_DOMAINS})"
         - "traefik.http.routers.${STACK_NAME}.entrypoints=web-secure"
         - "traefik.http.routers.${STACK_NAME}.tls.certresolver=${LETS_ENCRYPT_ENV}"
-        ## Redirect from EXTRA_DOMAINS to DOMAIN
+        - "coop-cloud.${STACK_NAME}.version=1.0.0+3.5.3"
-        #- "traefik.http.routers.${STACK_NAME}.middlewares=${STACK_NAME}-redirect"
-        #- "traefik.http.middlewares.${STACK_NAME}-redirect.headers.SSLForceHost=true"
-        #- "traefik.http.middlewares.${STACK_NAME}-redirect.headers.SSLHost=${DOMAIN}"
     healthcheck:
-      test: ["CMD", "curl", "-f", "http://localhost:3000"]
+      test: "curl -fsS http://localhost/srv/status || exit 1"
       interval: 30s
       timeout: 10s
-      retries: 10
+      retries: 6
-      start_period: 1m
+      start_period: 25m
 
   db:
-    image: postgres:11-alpine
+    image: pgvector/pgvector:pg17
     networks:
       - internal
     secrets:
       - db_password
     volumes:
       - 'postgresql_data:/var/lib/postgresql/data'
+    configs:
+      - source: db_entrypoint
+        target: /docker-entrypoint.sh
+        mode: 0555
+      - source: pg_backup
+        target: /pg_backup.sh
+        mode: 0555
+    entrypoint: /docker-entrypoint.sh
     environment:
       - POSTGRES_HOST_AUTH_METHOD=trust
       - POSTGRES_USER=discourse
       - POSTGRES_DB=discourse
       - POSTGRES_PASSWORD_FILE=/run/secrets/db_password
+    healthcheck:
+      test: "pg_isready -U discourse -d discourse"
+      interval: 30s
+      timeout: 10s
+      retries: 5
+      # generous: a postgres major-version upgrade (apt install + pg_upgrade) runs
+      # in the entrypoint before the server accepts connections — don't let the
+      # healthcheck kill an in-progress migration
+      start_period: 10m
+    deploy:
+      labels:
+        backupbot.backup: "true"
+        backupbot.backup.pre-hook: "/pg_backup.sh backup"
+        backupbot.backup.volumes.postgresql_data.path: "backup.sql"
+        backupbot.restore.post-hook: "/pg_backup.sh restore"
 
   redis:
-    image: redis:6.0-alpine
+    image: redis:7.4-alpine
     networks:
       - internal
     volumes:
       - 'redis_data:/data'
-
+    healthcheck:
-  sidekiq:
+      test: "redis-cli ping | grep -q PONG"
-    image: bitnami/discourse:2.7.7
+      interval: 30s
-    networks:
+      timeout: 5s
-      - internal
+      retries: 5
-    depends_on:
+      start_period: 30s
-      - discourse
-    volumes:
-      - 'sidekiq_data:/bitnami/discourse'
-    command: /opt/bitnami/scripts/discourse-sidekiq/run.sh
-    secrets:
-      - db_password
-    environment:
-      - ALLOW_EMPTY_PASSWORD=yes
-      - DISCOURSE_HOST=${DOMAIN}
-      - DISCOURSE_DATABASE_HOST=db
-      - DISCOURSE_DATABASE_PORT_NUMBER=5432
-      - DISCOURSE_DATABASE_USER=discourse
-      - DISCOURSE_DATABASE_NAME=discourse
-      - DISCOURSE_REDIS_HOST=redis
-      - DISCOURSE_REDIS_PORT_NUMBER=6379
-      - DISCOURSE_DATABASE_PASSWORD_FILE=/run/secrets/db_password
-      - DISCOURSE_SMTP_HOST
-      - DISCOURSE_SMTP_PORT
-      - DISCOURSE_SMTP_USER
-      - DISCOURSE_SMTP_PROTOCOL
-      - DISCOURSE_SMTP_AUTH
 
 secrets:
   db_password:
@@ -104,10 +120,28 @@ secrets:
 volumes:
   postgresql_data:
   redis_data:
+  discourse_shared:
   discourse_data:
-  sidekiq_data:
 
 networks:
   proxy:
     external: true
   internal:
+
+configs:
+  app_entrypoint:
+    name: ${STACK_NAME}_app_entrypoint_${APP_ENTRYPOINT_VERSION}
+    file: cc-app-entrypoint.sh
+  app_install_ssl:
+    name: ${STACK_NAME}_app_install_ssl_${APP_INSTALL_SSL_VERSION}
+    file: app-install-ssl.sh
+  app_migrate_uploads:
+    name: ${STACK_NAME}_app_migrate_uploads_${APP_MIGRATE_UPLOADS_VERSION}
+    file: migrate-uploads.sh
+  db_entrypoint:
+    name: ${STACK_NAME}_db_entrypoint_${DB_ENTRYPOINT_VERSION}
+    file: entrypoint.postgres.sh.tmpl
+    template_driver: golang
+  pg_backup:
+    name: ${STACK_NAME}_pg_backup_${PG_BACKUP_VERSION}
+    file: pg_backup.sh

entrypoint.postgres.sh.tmpl

@@ -0,0 +1,64 @@
+#!/bin/bash
+
+set -e
+
+OLDDATA=$PGDATA/old_data
+NEWDATA=$PGDATA/new_data
+
+echo "Running as $(id)"
+
+# The migration uses $OLDDATA/$NEWDATA as scratch and removes them when it
+# finishes; a leftover *empty* one means a run was interrupted before any data
+# moved (data still intact at $PGDATA) so we clear it and retry, while a
+# *non-empty* one means data may live only there, so we stop for manual recovery.
+for scratch in $OLDDATA $NEWDATA; do
+  if [ -d "$scratch" ] && [ -n "$(ls -A "$scratch")" ]; then
+    echo "FATAL: $scratch exists and is not empty - a previous migration did not"
+    echo "complete and the data may only exist there. manual recovery necessary."
+    exit 1
+  fi
+done
+rm -rf $OLDDATA $NEWDATA
+
+if [ -f $PGDATA/PG_VERSION ]; then
+  DATA_VERSION=$(cat $PGDATA/PG_VERSION)
+
+  if [ -n "$DATA_VERSION" -a "$PG_MAJOR" != "$DATA_VERSION" ]; then
+    echo "postgres data version $DATA_VERSION found, but need $PG_MAJOR. Starting migration"
+    echo "Installing postgres $DATA_VERSION"
+    sed -i "s/$/ $DATA_VERSION/" /etc/apt/sources.list.d/pgdg.list
+    apt-get update && apt-get install -y --no-install-recommends \
+      postgresql-$DATA_VERSION \
+      && rm -rf /var/lib/apt/lists/*
+    # pg_upgrade must run as the old cluster's bootstrap superuser (the "install
+    # user", oid 10), and the new cluster must be initialised with that same
+    # user. It is not necessarily $POSTGRES_USER (e.g. clusters created with the
+    # default "postgres" superuser and a separate app role), so read it from the
+    # old cluster: briefly start it and ask, connecting as the app role we know.
+    PGBIN=/usr/lib/postgresql/$DATA_VERSION/bin
+    gosu postgres $PGBIN/pg_ctl -D $PGDATA -w \
+      -o "-c listen_addresses= -c unix_socket_directories=/tmp" start
+    INSTALL_USER=$(gosu postgres psql -h /tmp -U "$POSTGRES_USER" -d postgres -tAc \
+      "select rolname from pg_roles where oid = 10")
+    gosu postgres $PGBIN/pg_ctl -D $PGDATA -w stop
+    echo "old cluster install user: $INSTALL_USER"
+    echo "shuffling around"
+    gosu postgres mkdir $OLDDATA $NEWDATA
+    chmod 700 $OLDDATA $NEWDATA
+    mv $PGDATA/* $OLDDATA/ || true
+    echo "running initdb"
+    # abuse entrypoint script for initdb by making server error out; initialise
+    # the new cluster with the same superuser as the old one so pg_upgrade matches
+    gosu postgres bash -c "export PGDATA=$NEWDATA POSTGRES_USER=$INSTALL_USER ; /usr/local/bin/docker-entrypoint.sh --invalid-arg || true"
+    echo "running pg_upgrade"
+    cd /tmp
+    gosu postgres pg_upgrade --link -b /usr/lib/postgresql/$DATA_VERSION/bin -d $OLDDATA -D $NEWDATA -U $INSTALL_USER
+    cp $OLDDATA/pg_hba.conf $NEWDATA/
+    mv $NEWDATA/* $PGDATA
+    rm -rf $OLDDATA
+    rmdir $NEWDATA
+    echo "migration complete"
+  fi
+fi
+
+/usr/local/bin/docker-entrypoint.sh postgres

migrate-uploads.sh

@@ -0,0 +1,24 @@
+#!/bin/bash
+# One-time, idempotent, NON-destructive migration of uploads + backups from a
+# legacy bitnami discourse volume into the official image's /shared.
+#
+# Runs on every boot as a runit 1.d hook but no-ops after the first success
+# (sentinel) and when there is no legacy volume mounted (fresh installs). It only
+# ever COPIES from the read-only /legacy mount, so an interruption just re-copies
+# on the next boot — there is no move/delete to leave the data half-migrated.
+set -e
+
+SENTINEL=/shared/.bitnami-uploads-migrated
+[ -e "$SENTINEL" ] && exit 0
+
+if [ -d /legacy/public/uploads ]; then
+  echo "[migrate-uploads] copying bitnami uploads/backups -> /shared"
+  mkdir -p /shared/uploads /shared/backups
+  cp -a /legacy/public/uploads/. /shared/uploads/ 2>/dev/null || true
+  cp -a /legacy/public/backups/. /shared/backups/ 2>/dev/null || true
+  # discourse runs as uid 1000; the official boot also chowns /shared, but be explicit
+  chown -R discourse:discourse /shared/uploads /shared/backups 2>/dev/null || true
+  echo "[migrate-uploads] done"
+fi
+
+touch "$SENTINEL"

pg_backup.sh

@@ -0,0 +1,44 @@
+#!/bin/bash
+
+# Postgres backup/restore hook for the discourse `db` service.
+
+set -e
+
+BACKUP_FILE='/var/lib/postgresql/data/backup.sql'
+export PGPASSWORD=$(cat "${POSTGRES_PASSWORD_FILE:-/run/secrets/db_password}")
+DB_USER="${POSTGRES_USER:-discourse}"
+DB_NAME="${POSTGRES_DB:-discourse}"
+
+function backup {
+  pg_dump -U "$DB_USER" "$DB_NAME" | gzip > "$BACKUP_FILE"
+}
+
+function restore {
+  cd /var/lib/postgresql/data/
+
+  # Block all non-local connections so the running discourse app + sidekiq cannot reconnect and
+  # interfere with the drop/recreate/reimport. Restored on exit.
+  restore_hba() {
+    cat pg_hba.conf.bak > pg_hba.conf
+    rm -f pg_hba.conf.bak
+    su postgres -c 'pg_ctl reload'
+  }
+  cp pg_hba.conf pg_hba.conf.bak
+  echo 'local all all trust' > pg_hba.conf
+  su postgres -c 'pg_ctl reload'
+  trap restore_hba EXIT INT TERM
+
+  # terminate any lingering local sessions before recreate
+  # see https://stackoverflow.com/questions/5108876/kill-a-postgresql-session-connection
+  psql -U "$DB_USER" -d postgres -c \
+    "SELECT pg_terminate_backend(pid) FROM pg_stat_activity WHERE datname='${DB_NAME}' AND pid<>pg_backend_pid();"
+
+  # drop database and then recreate it
+  psql -U "$DB_USER" -d postgres -c "DROP DATABASE ${DB_NAME} WITH (FORCE);"
+  createdb -U "$DB_USER" "$DB_NAME"
+
+  # reimport data 
+  gunzip -c "$BACKUP_FILE" | psql -U "$DB_USER" -d "$DB_NAME" -1 -v ON_ERROR_STOP=1 -f -
+}
+
+$@

release/1.0.0+3.5.3

@@ -0,0 +1,10 @@
+This release switches from the bitnami image to the official discourse/discourse
+image. Some env vars need to be renamed for this migration; everything else
+should happen automatically.
+
+Rename these in your app's .env (the values carry over):
+
+  DISCOURSE_SMTP_HOST     --> DISCOURSE_SMTP_ADDRESS
+  DISCOURSE_SMTP_USER     --> DISCOURSE_SMTP_USER_NAME
+  DISCOURSE_SMTP_AUTH     --> DISCOURSE_SMTP_AUTHENTICATION
+  DISCOURSE_SMTP_PROTOCOL --> DISCOURSE_SMTP_ENABLE_START_TLS (takes a boolean true/false, not the old tls/ssl value, so translate it rather than copying it straight across)