chore: upgrade impress to v6.0.2 (yhub collaboration), nginx to 1.31.8, docspec 1.21.4 #11

Open
autonomic-bot wants to merge 2 commits from upgrade-v6.0.2 into main
4 changed files with 61 additions and 51 deletions
Showing only changes of commit ae725c7e00 - Show all commits
-4
View File
@@ -27,10 +27,6 @@ SECRET_POSTGRES_P_VERSION=v1
SECRET_Y_API_KEY_VERSION=v1 SECRET_Y_API_KEY_VERSION=v1
# DJANGO_HOST_EMAIL_PASSWORD # DJANGO_HOST_EMAIL_PASSWORD
SECRET_EMAIL_PASS_VERSION=v1 SECRET_EMAIL_PASS_VERSION=v1
# JWT_PRIVATE_KEY of the backend (RSA PEM, must be inserted manually, see README)
SECRET_JWT_PK_VERSION=v1 # generate=false
# YHUB_JWT_PRIVATE_KEY of the collaboration server (RSA PEM, must be inserted manually, see README)
SECRET_YHUB_JWT_PK_VERSION=v1 # generate=false
############################################################################## ##############################################################################
# EMAIL # EMAIL
+2 -11
View File
@@ -19,7 +19,6 @@
## Quick start ## Quick start
* `abra app new lasuite-docs --secrets` * `abra app new lasuite-docs --secrets`
* Insert the two RSA keys, which abra cannot generate (see below)
* `abra app config <app-name>` * `abra app config <app-name>`
* `abra app deploy <app-name>` * `abra app deploy <app-name>`
@@ -30,21 +29,13 @@ You should then be able to visit the landing page of your app, but not yet to lo
## RSA keys ## RSA keys
Since v6, the backend and the collaboration server (`yhub`) authenticate each other with signed tokens. Each needs its own RSA private key, never the same one: Since v6, the backend and the collaboration server (`yhub`) authenticate each other with signed tokens. Each needs its own RSA private key, never the same one. The recipe generates both automatically on first deploy into the `jwt_keys` volume (the `jwt-keys` service, using the yhub image's Node crypto) and reuses them on later deploys. They are not Swarm secrets because abra cannot generate PEM keys and Swarm secrets must exist before the stack is deployed.
```
openssl genpkey -algorithm RSA -pkeyopt rsa_keygen_bits:2048 -out jwt.pem
openssl genpkey -algorithm RSA -pkeyopt rsa_keygen_bits:2048 -out yhub-jwt.pem
abra app secret insert <app-name> jwt_pk v1 jwt.pem -f
abra app secret insert <app-name> yhub_jwt_pk v1 yhub-jwt.pem -f
rm jwt.pem yhub-jwt.pem
```
## Upgrading from v5 to v6 ## Upgrading from v5 to v6
v6 replaces the collaboration server. Document content moves from the minio bucket to the new `yhub` service, which stores it in its own `yhub` database. If you skip the migration, every existing document opens **empty**. See [UPGRADE.md](https://github.com/suitenumerique/docs/blob/main/UPGRADE.md) upstream for details. v6 replaces the collaboration server. Document content moves from the minio bucket to the new `yhub` service, which stores it in its own `yhub` database. If you skip the migration, every existing document opens **empty**. See [UPGRADE.md](https://github.com/suitenumerique/docs/blob/main/UPGRADE.md) upstream for details.
1. Insert the two RSA keys (see "RSA keys" above). The `collab_ss` secret is no longer used. 1. The `collab_ss` secret is no longer used. The two RSA keys are generated automatically on first deploy (see "RSA keys" above).
2. Leave `SOFT_MIGRATION` at its default (`true`) and deploy. The `yhub` schema is created on startup. A document is copied to yhub the first time someone opens it. 2. Leave `SOFT_MIGRATION` at its default (`true`) and deploy. The `yhub` schema is created on startup. A document is copied to yhub the first time someone opens it.
3. Copy all remaining documents: `abra app cmd <app-name> backend migrate_documents`. You can pass arguments, e.g. `abra app cmd <app-name> backend migrate_documents -- --dry-run`. Re-run it with `--retry-failed` if some documents failed. 3. Copy all remaining documents: `abra app cmd <app-name> backend migrate_documents`. You can pass arguments, e.g. `abra app cmd <app-name> backend migrate_documents -- --dry-run`. Re-run it with `--retry-failed` if some documents failed.
4. Only after that, you may set `SOFT_MIGRATION=false`. Keep the minio bucket and its object versions: they hold the document history from before v6. 4. Only after that, you may set `SOFT_MIGRATION=false`. Keep the minio bucket and its object versions: they hold the document history from before v6.
+55 -23
View File
@@ -57,7 +57,7 @@ x-common-env: &common-env
# Collaboration server (yhub), reached in-stack # Collaboration server (yhub), reached in-stack
YHUB_API_BASE_URL: http://yhub:3002 YHUB_API_BASE_URL: http://yhub:3002
# RSA key signing the tokens the backend presents to yhub and y-provider # RSA key signing the tokens the backend presents to yhub and y-provider
JWT_PRIVATE_KEY_FILE: /run/secrets/jwt_pk JWT_PRIVATE_KEY_FILE: /keys/jwt.pem
# Email app URL (used in email templates; matches recipe DOMAIN) # Email app URL (used in email templates; matches recipe DOMAIN)
DJANGO_EMAIL_URL_APP: "https://${DOMAIN}" DJANGO_EMAIL_URL_APP: "https://${DOMAIN}"
# Conversion (file upload → Yjs via y-provider) # Conversion (file upload → Yjs via y-provider)
@@ -72,7 +72,7 @@ x-common-env: &common-env
Y_PROVIDER_API_BASE_URL: http://y-provider:4444/api/ Y_PROVIDER_API_BASE_URL: http://y-provider:4444/api/
Y_PROVIDER_API_KEY_FILE: /run/secrets/y_api_key Y_PROVIDER_API_KEY_FILE: /run/secrets/y_api_key
# DocSpec — converts .docx → BlockNote JSON, then y-provider turns that into Yjs # DocSpec — converts .docx → BlockNote JSON, then y-provider turns that into Yjs
DOCSPEC_API_URL: http://docspec:4000/conversion DOCSPEC_API_URL: http://docspec:3000/conversion
# Database connection pool # Database connection pool
DB_PSYCOPG_POOL_ENABLED: DB_PSYCOPG_POOL_ENABLED:
# Media auth header (matches recipe nginx by default) # Media auth header (matches recipe nginx by default)
@@ -105,7 +105,7 @@ x-yhub-env: &yhub-env
COLLABORATION_BACKEND_BASE_URL: https://${DOMAIN} COLLABORATION_BACKEND_BASE_URL: https://${DOMAIN}
COLLABORATION_SERVER_ORIGIN: https://${DOMAIN} COLLABORATION_SERVER_ORIGIN: https://${DOMAIN}
Y_PROVIDER_API_KEY_FILE: /run/secrets/y_api_key Y_PROVIDER_API_KEY_FILE: /run/secrets/y_api_key
YHUB_JWT_PRIVATE_KEY_FILE: /run/secrets/yhub_jwt_pk YHUB_JWT_PRIVATE_KEY_FILE: /keys/yhub-jwt.pem
# Seed documents from the pre-v6 media bucket the first time they are opened. # Seed documents from the pre-v6 media bucket the first time they are opened.
# Only turn off once `abra app cmd <app> backend migrate_documents` covered all documents. # Only turn off once `abra app cmd <app> backend migrate_documents` covered all documents.
SOFT_MIGRATION: "${SOFT_MIGRATION:-true}" SOFT_MIGRATION: "${SOFT_MIGRATION:-true}"
@@ -120,7 +120,7 @@ x-minio-env: &minio-env
services: services:
app: app:
image: lasuite/impress-frontend:v6.0.1 image: lasuite/impress-frontend:v6.0.2
networks: networks:
- backend - backend
deploy: deploy:
@@ -137,7 +137,7 @@ services:
start_period: 10s start_period: 10s
backend: backend:
image: lasuite/impress-backend:v6.0.1 image: lasuite/impress-backend:v6.0.2
networks: networks:
- backend - backend
environment: environment:
@@ -152,7 +152,9 @@ services:
user: "${DOCKER_USER:-1000}" user: "${DOCKER_USER:-1000}"
command: ["gunicorn", "-c", "/usr/local/etc/gunicorn/impress.py", "impress.wsgi:application"] command: ["gunicorn", "-c", "/usr/local/etc/gunicorn/impress.py", "impress.wsgi:application"]
entrypoint: > entrypoint: >
sh -c "if [ \"$$AUTO_MIGRATIONS\" = \"true\" ]; then /migrate.sh; fi && exec /abra-entrypoint.sh /usr/local/bin/entrypoint \"$$@\"" -- sh -c "while [ ! -s /keys/jwt.pem ]; do sleep 1; done; if [ \"$$AUTO_MIGRATIONS\" = \"true\" ]; then /migrate.sh; fi && exec /abra-entrypoint.sh /usr/local/bin/entrypoint \"$$@\"" --
volumes:
- jwt_keys:/keys:ro
configs: configs:
- source: abra_entrypoint - source: abra_entrypoint
target: /abra-entrypoint.sh target: /abra-entrypoint.sh
@@ -169,10 +171,9 @@ services:
- postgres_p - postgres_p
- y_api_key - y_api_key
- email_pass - email_pass
- jwt_pk
celery: celery:
image: lasuite/impress-backend:v6.0.1 image: lasuite/impress-backend:v6.0.2
networks: networks:
- backend - backend
healthcheck: healthcheck:
@@ -185,7 +186,10 @@ services:
command: ["celery", "-A", "impress.celery_app", "worker", "-l", "INFO"] command: ["celery", "-A", "impress.celery_app", "worker", "-l", "INFO"]
environment: environment:
<<: [*common-env, *postgres-env, *yprovider-env] <<: [*common-env, *postgres-env, *yprovider-env]
entrypoint: ["/abra-entrypoint.sh", "/usr/local/bin/entrypoint"] entrypoint: >
sh -c "while [ ! -s /keys/jwt.pem ]; do sleep 1; done; exec /abra-entrypoint.sh /usr/local/bin/entrypoint \"$$@\"" --
volumes:
- jwt_keys:/keys:ro
configs: configs:
- source: abra_entrypoint - source: abra_entrypoint
target: /abra-entrypoint.sh target: /abra-entrypoint.sh
@@ -199,11 +203,10 @@ services:
- postgres_p - postgres_p
- y_api_key - y_api_key
- email_pass - email_pass
- jwt_pk
# Collaboration server: serves the websocket and holds the document content # Collaboration server: serves the websocket and holds the document content
yhub: yhub:
image: lasuite/impress-yhub:v6.0.1 image: lasuite/impress-yhub:v6.0.2
networks: networks:
- backend - backend
environment: *yhub-env environment: *yhub-env
@@ -216,12 +219,13 @@ services:
start_period: 30s start_period: 30s
# The server never creates its schema: run the idempotent init-db before every start # The server never creates its schema: run the idempotent init-db before every start
entrypoint: > entrypoint: >
sh -c "export POSTGRES=\"postgres://docs:$$(node -e 'process.stdout.write(encodeURIComponent(require(\"fs\").readFileSync(\"/run/secrets/postgres_p\",\"utf8\").trim()))')@db:5432/yhub\" && node node_modules/@y/hub/bin/init-db.js && exec /usr/local/bin/entrypoint \"$$@\"" -- sh -c "while [ ! -s /keys/yhub-jwt.pem ]; do sleep 1; done; export POSTGRES=\"postgres://docs:$$(node -e 'process.stdout.write(encodeURIComponent(require(\"fs\").readFileSync(\"/run/secrets/postgres_p\",\"utf8\").trim()))')@db:5432/yhub\" && node node_modules/@y/hub/bin/init-db.js && exec /usr/local/bin/entrypoint \"$$@\"" --
command: ["node", "--import", "./dist/sentry.js", "dist/server.js"] command: ["node", "--import", "./dist/sentry.js", "dist/server.js"]
volumes:
- jwt_keys:/keys:ro
secrets: secrets:
- postgres_p - postgres_p
- y_api_key - y_api_key
- yhub_jwt_pk
- minio_ru - minio_ru
- minio_rp - minio_rp
@@ -242,9 +246,42 @@ services:
labels: labels:
backupbot.backup: "${ENABLE_BACKUPS:-true}" backupbot.backup: "${ENABLE_BACKUPS:-true}"
# Generates the two RSA keys the backend and yhub sign with, into a shared volume.
# They cannot be abra-generated secrets (PEM format) and Swarm secrets cannot be
# created at deploy time, so this idempotent sidecar creates them once and keeps them
# on the volume across redeploys. Both consumers wait for their file before starting.
jwt-keys:
image: lasuite/impress-yhub:v6.0.2
networks:
- backend
user: "0"
volumes:
- jwt_keys:/keys
healthcheck:
test: ["CMD-SHELL", "[ -s /keys/jwt.pem ] && [ -s /keys/yhub-jwt.pem ]"]
interval: 15s
timeout: 5s
retries: 3
start_period: 10s
entrypoint: ["/bin/sh", "-c"]
command:
- |
set -eu
generate() {
target="/keys/$$1"
if [ ! -s "$$target" ]; then
node -e 'const fs=require("fs");const {generateKeyPairSync}=require("crypto");const {privateKey}=generateKeyPairSync("rsa",{modulusLength:2048,publicKeyEncoding:{type:"spki",format:"pem"},privateKeyEncoding:{type:"pkcs8",format:"pem"}});fs.writeFileSync(process.argv[1],privateKey,{mode:0o400});' "$$target"
fi
}
generate jwt.pem
generate yhub-jwt.pem
chown -R "${DOCKER_USER:-1000}:${DOCKER_USER:-1000}" /keys
chmod 0400 /keys/jwt.pem /keys/yhub-jwt.pem
while true; do sleep 3600; done
# Conversion service only since v6 (no longer serves the collaboration) # Conversion service only since v6 (no longer serves the collaboration)
y-provider: y-provider:
image: lasuite/impress-y-provider:v6.0.1 image: lasuite/impress-y-provider:v6.0.2
networks: networks:
- backend - backend
healthcheck: healthcheck:
@@ -263,12 +300,12 @@ services:
- y_api_key - y_api_key
docspec: docspec:
image: ghcr.io/docspecio/api:3.0.2 image: ghcr.io/docspec/api:1.21.4
networks: networks:
- backend - backend
healthcheck: healthcheck:
# Use 127.0.0.1, which is required instead of localhost for this healtcheck because of elixir/busybox bindings # Use 127.0.0.1, which is required instead of localhost for this healtcheck because of elixir/busybox bindings
test: ["CMD", "wget", "-q", "-O", "/dev/null", "http://127.0.0.1:4000/health"] test: ["CMD", "wget", "-q", "-O", "/dev/null", "http://127.0.0.1:3000/health"]
interval: 15s interval: 15s
timeout: 5s timeout: 5s
retries: 3 retries: 3
@@ -343,7 +380,7 @@ services:
- minio_ru - minio_ru
web: web:
image: nginx:1.31.7 image: nginx:1.31.8
healthcheck: healthcheck:
test: ["CMD", "curl", "-f", "http://localhost:8083"] test: ["CMD", "curl", "-f", "http://localhost:8083"]
interval: 15s interval: 15s
@@ -380,6 +417,7 @@ volumes:
postgres: postgres:
minio: minio:
yhub_valkey: yhub_valkey:
jwt_keys:
configs: configs:
nginx_conf: nginx_conf:
@@ -412,12 +450,6 @@ secrets:
postgres_p: postgres_p:
external: true external: true
name: ${STACK_NAME}_postgres_p_${SECRET_POSTGRES_P_VERSION} name: ${STACK_NAME}_postgres_p_${SECRET_POSTGRES_P_VERSION}
jwt_pk:
external: true
name: ${STACK_NAME}_jwt_pk_${SECRET_JWT_PK_VERSION}
yhub_jwt_pk:
external: true
name: ${STACK_NAME}_yhub_jwt_pk_${SECRET_YHUB_JWT_PK_VERSION}
minio_rp: minio_rp:
external: true external: true
name: ${STACK_NAME}_minio_rp_${SECRET_MINIO_RP_VERSION} name: ${STACK_NAME}_minio_rp_${SECRET_MINIO_RP_VERSION}
+4 -13
View File
@@ -1,19 +1,10 @@
⚠️ Major upgrade to Docs v6: the collaboration server is replaced by the new `yhub` ⚠️ Major upgrade to Docs v6: the collaboration server is replaced by the new `yhub`
service, and existing documents have to be migrated to it, or they open EMPTY. service, and existing documents have to be migrated to it, or they open EMPTY.
Before deploying, generate and insert two RSA keys: `SECRET_COLLAB_SS_VERSION` and the `collab_ss` secret are no longer used. The two
RSA keys the backend and yhub sign with are generated automatically on first
openssl genpkey -algorithm RSA -pkeyopt rsa_keygen_bits:2048 -out jwt.pem deploy into the `jwt_keys` volume (the `jwt-keys` service) and reused afterwards;
openssl genpkey -algorithm RSA -pkeyopt rsa_keygen_bits:2048 -out yhub-jwt.pem no manual secret insertion is needed.
abra app secret insert <app-name> jwt_pk v1 jwt.pem -f
abra app secret insert <app-name> yhub_jwt_pk v1 yhub-jwt.pem -f
and add to your app config:
SECRET_JWT_PK_VERSION=v1 # generate=false
SECRET_YHUB_JWT_PK_VERSION=v1 # generate=false
`SECRET_COLLAB_SS_VERSION` and the `collab_ss` secret are no longer used.
After deploying (SOFT_MIGRATION is on by default), migrate all documents: After deploying (SOFT_MIGRATION is on by default), migrate all documents: