From 6ee7a426861bc377860a330177133b75f5e4de05 Mon Sep 17 00:00:00 2001 From: 3wc <3wc.cyberia@doesthisthing.work> Date: Tue, 20 Jul 2021 11:34:56 +0200 Subject: [PATCH] Split README up into separate files --- README.md | 230 +----------------- docs/architecture.md | 26 ++ docs/btcpay.md | 68 ++++++ docs/configuration.md | 23 ++ docs/database.md | 46 ++++ docs/deployment.md | 68 ++++++ .../images}/btcpay_sin_pairing.jpg | Bin .../images}/btcpayment_process.drawio | 0 .../images}/btcpayment_process.png | Bin .../images}/generate_btcpay_keys.py | 0 {readme => docs/images}/hub-and-spoke.xml | 0 {readme => docs/images}/hub-and-spoke1.png | Bin {readme => docs/images}/hub-and-spoke2.png | Bin {readme => docs/images}/paired.jpg | Bin docs/local-set-up.md | 69 ++++++ 15 files changed, 307 insertions(+), 223 deletions(-) create mode 100644 docs/architecture.md create mode 100644 docs/btcpay.md create mode 100644 docs/configuration.md create mode 100644 docs/database.md create mode 100644 docs/deployment.md rename {readme => docs/images}/btcpay_sin_pairing.jpg (100%) rename {readme => docs/images}/btcpayment_process.drawio (100%) rename {readme => docs/images}/btcpayment_process.png (100%) rename {readme => docs/images}/generate_btcpay_keys.py (100%) rename {readme => docs/images}/hub-and-spoke.xml (100%) rename {readme => docs/images}/hub-and-spoke1.png (100%) rename {readme => docs/images}/hub-and-spoke2.png (100%) rename {readme => docs/images}/paired.jpg (100%) create mode 100644 docs/local-set-up.md diff --git a/README.md b/README.md index 1442a7d..f60195e 100644 --- a/README.md +++ b/README.md @@ -2,226 +2,10 @@ Python Flask web application for capsul.org - -## how to run locally - -Ensure you have the pre-requisites for the psycopg2 Postgres database adapter package - -``` -sudo apt install python3-dev libpq-dev -pg_config --version -``` - -Ensure you have the wonderful `pipenv` python package management and virtual environment cli - -``` -sudo apt install pipenv -``` - -Create python virtual environment and install packages - -``` -# install deps -pipenv install -``` - -Run an instance of Postgres (I used docker for this, you can use whatever you want, point is its listening on localhost:5432) - -``` -docker run --rm -it -e POSTGRES_PASSWORD=dev -p 5432:5432 postgres -``` - -Run the app - -``` -pipenv run flask run -``` - -Run the app in gunicorn: - -``` -pipenv run gunicorn --bind 127.0.0.1:5000 -k gevent --worker-connections 1000 app:app -``` - -Once you log in for the first time, you will want to give yourself some free capsulbux so you can create fake capsuls for testing. - -Note that by default when running locally, the `SPOKE_MODEL` is set to `mock`, meaning that it won't actually try to spawn vms. - -``` -pipenv run flask cli sql -c "INSERT INTO payments (email, dollars) VALUES ('', 20.00)" -``` - -## configuration: - -Create a `.env` file to set up the application configuration: - -``` -nano .env -``` - -You can enter any environment variables referenced in `__init__.py` to this file. - -For example you may enter your SMTP credentials like this: -``` -MAIL_USERNAME=forest@nullhex.com -MAIL_DEFAULT_SENDER=forest@nullhex.com -MAIL_PASSWORD=************** -``` - -## how to view the logs on the database server (legion.cyberia.club) - -`sudo -u postgres pg_dump capsul-flask | gzip -9 > capsul-backup-2021-02-15.gz` - ------ - -## cli - -You can manually mess around with the database like this: - -``` -pipenv run flask cli sql -f test.sql -``` - -``` -pipenv run flask cli sql -c 'SELECT * FROM vms' -``` - -This one selects the vms table with the column name header: - -``` -pipenv run flask cli sql -c "SELECT string_agg(column_name::text, ', ') from information_schema.columns WHERE table_name='vms'; SELECT * from vms" -``` - -How to modify a payment manually, like if you get a chargeback or to fix customer payment issues: - -``` -$ pipenv run flask cli sql -c "SELECT id, created, email, dollars, invalidated from payments" -1, 2020-05-05T00:00:00, forest.n.johnson@gmail.com, 20.00, FALSE - -$ pipenv run flask cli sql -c "UPDATE payments SET invalidated = True WHERE id = 1" -1 rows affected. - -$ pipenv run flask cli sql -c "SELECT id, created, email, dollars, invalidated from payments" -1, 2020-05-05T00:00:00, forest.n.johnson@gmail.com, 20.00, TRUE -``` - - -How you would kick off the scheduled task: - -``` -pipenv run flask cli cron-task -``` - ------ - -## postgres database schema management - -capsulflask has a concept of a schema version. When the application starts, it will query the database for a table named -`schemaversion` that has one row and one column (`version`). If the `version` it finds is not equal to the `desiredSchemaVersion` variable set in `db.py`, it will run migration scripts from the `schema_migrations` folder one by one until the `schemaversion` table shows the correct version. - -For example, the script named `02_up_xyz.sql` should contain code that migrates the database from schema version 1 to schema version 2. Likewise, the script `02_down_xyz.sql` should contain code that migrates from schema version 2 back to schema version 1. - -**IMPORTANT: if you need to make changes to the schema, make a NEW schema version. DO NOT EDIT the existing schema versions.** - -In general, for safety, schema version upgrades should not delete data. Schema version downgrades will simply throw an error and exit for now. - ------ - -## hub-and-spoke architecture - -![](readme/hub-and-spoke1.png) - -This diagram was created with https://app.diagrams.net/. -To edit it, download the diagram file and edit it with the https://app.diagrams.net/ web application, or you may run the application from [source](https://github.com/jgraph/drawio) if you wish. - -right now I have 2 types of operations, immediate mode and async. - -both types of operations do assignment synchronously. so if the system cant assign the operation to one or more hosts (spokes), -or whatever the operation requires, then it will fail. - -some operations tolerate partial failures, like, `capacity_avaliable` will succeed if at least one spoke succeeds. -for immediate mode requests (like `list`, `capacity_avaliable`, `destroy`), assignment and completion of the operation are the same thing. - -for async ones, they can be assigned without knowing whether or not they succeeded (`create`). - -![](readme/hub-and-spoke2.png) - -This diagram was created with https://app.diagrams.net/. -To edit it, download the diagram file and edit it with the https://app.diagrams.net/ web application, or you may run the application from [source](https://github.com/jgraph/drawio) if you wish. - -if you issue a create, and it technically could go to any number of hosts, but only one host responds, it will succeed -but if you issue a create and somehow 2 hosts both think they own that task, it will fail and throw a big error. cuz it expects exactly 1 to own the create task - -currently its not set up to do any polling. its not really like a queue at all. It's all immediate for the most part - ------ - -## how to setup btcpay server - -Generate a private key and the accompanying bitpay SIN for the btcpay API client. - - I used this code as an example: https://github.com/bitpay/bitpay-python/blob/master/bitpay/key_utils.py#L6 - -``` -$ pipenv run python ./readme/generate_btcpay_keys.py -``` - -It should output something looking like this: - -``` ------BEGIN EC PRIVATE KEY----- -EXAMPLEIArx/EXAMPLEKH23EXAMPLEsYXEXAMPLE5qdEXAMPLEcFHoAcEXAMPLEK -oUQDQgAEnWs47PT8+ihhzyvXX6/yYMAWWODluRTR2Ix6ZY7Z+MV7v0W1maJzqeqq -NQ+cpBvPDbyrDk9+Uf/sEaRCma094g== ------END EC PRIVATE KEY----- - - -EXAMPLEwzAEXAMPLEEXAMPLEURD7EXAMPLE -``` - -In order to register the key with the btcpay server, you have to first generate a pairing token using the btcpay server interface. -This requires your btcpay server account to have access to the capsul store. Ask Cass about this. - -Navigate to `Manage store: Access Tokens` at: `https://btcpay.cyberia.club/stores//Tokens` - -![](readme/btcpay_sin_pairing.jpg) - - -Finally, send an http request to the btcpay server to complete the pairing: - -``` -curl -H "Content-Type: application/json" https://btcpay.cyberia.club/tokens -d "{'id': 'EXAMPLEwzAEXAMPLEEXAMPLEURD7EXAMPLE', 'pairingCode': 'XXXXXXX'}" -``` - -It should respond with a token: - -``` -{"data":[{"policies":[],"pairingCode":"XXXXXXX","pairingExpiration":1589473817597,"dateCreated":1589472917597,"facade":"merchant","token":"xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx","label":"capsulflask"}]} -``` - -And you should see the token in the btcpay server UI: - -![](readme/paired.jpg) - -Now simply set your `BTCPAY_PRIVATE_KEY` variable in `.env` - -NOTE: make sure to use single quotes and replace the new lines with \n. - -``` -BTCPAY_PRIVATE_KEY='-----BEGIN EC PRIVATE KEY-----\nEXAMPLEIArx/EXAMPLEKH23EXAMPLEsYXEXAMPLE5qdEXAMPLEcFHoAcEXAMPLEK\noUQDQgAEnWs47PT8+ihhzyvXX6/yYMAWWODluRTR2Ix6ZY7Z+MV7v0W1maJzqeqq\nNQ+cpBvPDbyrDk9+Uf/sEaRCma094g==\n-----END EC PRIVATE KEY-----' -``` - - ------ - -## testing cryptocurrency payments - -I used litecoin to test cryptocurrency payments, because its the simplest & lowest fee cryptocurrency that BTCPay server supports. You can download the easy-to-use litecoin SPV wallet `electrum-ltc` from [github.com/pooler/electrum-ltc](https://github.com/pooler/electrum-ltc) or [electrum-ltc.org](https://electrum-ltc.org/), set up a wallet, and then either purchase some litecoin from an exchange, or ask Forest for some litecoin to use for testing. - - -## sequence diagram explaining how BTC payment process works - -![btcpayment_process](readme/btcpayment_process.png) - -This diagram was created with https://app.diagrams.net/. -To edit it, download the diagram file and edit it with the https://app.diagrams.net/ web application, or you may run the application from [source](https://github.com/jgraph/drawio) if you wish. \ No newline at end of file +How about a trip to the the `docs/` folder? +- [Setting up Capsul locally](./docs/local-set-up.md) +- [Hub-and-spoke architecture](./docs/architecture.md) +- [Deplying Capsul on a server](./docs/deployment.md) +- [Configuring Capsul](./docs/configuration.md) +- [Receiving cryptocurrency payments with BTCPay](./docs/btcpay.md) +- [Working with the database](./docs/database.md) diff --git a/docs/architecture.md b/docs/architecture.md new file mode 100644 index 0000000..a385c7c --- /dev/null +++ b/docs/architecture.md @@ -0,0 +1,26 @@ +# hub-and-spoke architecture + +![](images/hub-and-spoke1.png) + +This diagram was created with https://app.diagrams.net/. +To edit it, download the diagram file and edit it with the https://app.diagrams.net/ web application, or you may run the application from [source](https://github.com/jgraph/drawio) if you wish. + +right now I have 2 types of operations, immediate mode and async. + +both types of operations do assignment synchronously. so if the system cant assign the operation to one or more hosts (spokes), +or whatever the operation requires, then it will fail. + +some operations tolerate partial failures, like, `capacity_avaliable` will succeed if at least one spoke succeeds. +for immediate mode requests (like `list`, `capacity_avaliable`, `destroy`), assignment and completion of the operation are the same thing. + +for async ones, they can be assigned without knowing whether or not they succeeded (`create`). + +![](images/hub-and-spoke2.png) + +This diagram was created with https://app.diagrams.net/. +To edit it, download the diagram file and edit it with the https://app.diagrams.net/ web application, or you may run the application from [source](https://github.com/jgraph/drawio) if you wish. + +if you issue a create, and it technically could go to any number of hosts, but only one host responds, it will succeed +but if you issue a create and somehow 2 hosts both think they own that task, it will fail and throw a big error. cuz it expects exactly 1 to own the create task + +currently its not set up to do any polling. its not really like a queue at all. It's all immediate for the most part diff --git a/docs/btcpay.md b/docs/btcpay.md new file mode 100644 index 0000000..ced7108 --- /dev/null +++ b/docs/btcpay.md @@ -0,0 +1,68 @@ +# Receiving cryptocurrency payments with BTCPay + +Generate a private key and the accompanying bitpay SIN for the btcpay API client. + +I used this code as an example: https://github.com/bitpay/bitpay-python/blob/master/bitpay/key_utils.py#L6 + +``` +$ pipenv run python ./readme/generate_btcpay_keys.py +``` + +It should output something looking like this: + +``` +-----BEGIN EC PRIVATE KEY----- +EXAMPLEIArx/EXAMPLEKH23EXAMPLEsYXEXAMPLE5qdEXAMPLEcFHoAcEXAMPLEK +oUQDQgAEnWs47PT8+ihhzyvXX6/yYMAWWODluRTR2Ix6ZY7Z+MV7v0W1maJzqeqq +NQ+cpBvPDbyrDk9+Uf/sEaRCma094g== +-----END EC PRIVATE KEY----- + + +EXAMPLEwzAEXAMPLEEXAMPLEURD7EXAMPLE +``` + +In order to register the key with the btcpay server, you have to first generate a pairing token using the btcpay server interface. +This requires your btcpay server account to have access to the capsul store. Ask Cass about this. + +Navigate to `Manage store: Access Tokens` at: `https://btcpay.cyberia.club/stores//Tokens` + +![](readme/btcpay_sin_pairing.jpg) + + +Finally, send an http request to the btcpay server to complete the pairing: + +``` +curl -H "Content-Type: application/json" https://btcpay.cyberia.club/tokens -d "{'id': 'EXAMPLEwzAEXAMPLEEXAMPLEURD7EXAMPLE', 'pairingCode': 'XXXXXXX'}" +``` + +It should respond with a token: + +``` +{"data":[{"policies":[],"pairingCode":"XXXXXXX","pairingExpiration":1589473817597,"dateCreated":1589472917597,"facade":"merchant","token":"xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx","label":"capsulflask"}]} +``` + +And you should see the token in the btcpay server UI: + +![](readme/paired.jpg) + +Now simply set your `BTCPAY_PRIVATE_KEY` variable in `.env` + +NOTE: make sure to use single quotes and replace the new lines with \n. + +``` +BTCPAY_PRIVATE_KEY='-----BEGIN EC PRIVATE KEY-----\nEXAMPLEIArx/EXAMPLEKH23EXAMPLEsYXEXAMPLE5qdEXAMPLEcFHoAcEXAMPLEK\noUQDQgAEnWs47PT8+ihhzyvXX6/yYMAWWODluRTR2Ix6ZY7Z+MV7v0W1maJzqeqq\nNQ+cpBvPDbyrDk9+Uf/sEaRCma094g==\n-----END EC PRIVATE KEY-----' +``` + +----- + +## testing cryptocurrency payments + +I used litecoin to test cryptocurrency payments, because its the simplest & lowest fee cryptocurrency that BTCPay server supports. You can download the easy-to-use litecoin SPV wallet `electrum-ltc` from [github.com/pooler/electrum-ltc](https://github.com/pooler/electrum-ltc) or [electrum-ltc.org](https://electrum-ltc.org/), set up a wallet, and then either purchase some litecoin from an exchange, or ask Forest for some litecoin to use for testing. + + +## sequence diagram explaining how BTC payment process works + +![btcpayment_process](readme/btcpayment_process.png) + +This diagram was created with https://app.diagrams.net/. +To edit it, download the diagram file and edit it with the https://app.diagrams.net/ web application, or you may run the application from [source](https://github.com/jgraph/drawio) if you wish. diff --git a/docs/configuration.md b/docs/configuration.md new file mode 100644 index 0000000..26b764f --- /dev/null +++ b/docs/configuration.md @@ -0,0 +1,23 @@ +# Configuring Capsul-Flask + +Create a `.env` file to set up the application configuration: + +``` +nano .env +``` + +You can enter any environment variables referenced in `__init__.py` to this file. + +For example you may enter your SMTP credentials like this: +``` +MAIL_USERNAME=forest@nullhex.com +MAIL_DEFAULT_SENDER=forest@nullhex.com +MAIL_PASSWORD=************** +``` + +## Loading variables from files + +To support [Docker Secrets](https://docs.docker.com/engine/swarm/secrets/), you can also load secret values from files – for example, to load `MAIL_PASSWORD` from `/run/secrets/mail_password`, set +```sh +MAIL_PASSWORD_FILE=/run/secrets/mail_password +``` diff --git a/docs/database.md b/docs/database.md new file mode 100644 index 0000000..b2b9d72 --- /dev/null +++ b/docs/database.md @@ -0,0 +1,46 @@ +# Working with the Capsul database + +## Running manual database queries + +You can manually mess around with the database like this: + +``` +pipenv run flask cli sql -f test.sql +``` + +``` +pipenv run flask cli sql -c 'SELECT * FROM vms' +``` + +This one selects the vms table with the column name header: + +``` +pipenv run flask cli sql -c "SELECT string_agg(column_name::text, ', ') from information_schema.columns WHERE table_name='vms'; SELECT * from vms" +``` + +How to modify a payment manually, like if you get a chargeback or to fix customer payment issues: + +``` +$ pipenv run flask cli sql -c "SELECT id, created, email, dollars, invalidated from payments" +1, 2020-05-05T00:00:00, forest.n.johnson@gmail.com, 20.00, FALSE + +$ pipenv run flask cli sql -c "UPDATE payments SET invalidated = True WHERE id = 1" +1 rows affected. + +$ pipenv run flask cli sql -c "SELECT id, created, email, dollars, invalidated from payments" +1, 2020-05-05T00:00:00, forest.n.johnson@gmail.com, 20.00, TRUE +``` + +## Database schema management + +capsulflask has a concept of a schema version. When the application starts, it will query the database for a table named `schemaversion` that has one row and one column (`version`). If the `version` it finds is not equal to the `desiredSchemaVersion` variable set in `db.py`, it will run migration scripts from the `schema_migrations` folder one by one until the `schemaversion` table shows the correct version. + +For example, the script named `02_up_xyz.sql` should contain code that migrates the database from schema version 1 to schema version 2. Likewise, the script `02_down_xyz.sql` should contain code that migrates from schema version 2 back to schema version 1. + +**IMPORTANT: if you need to make changes to the schema, make a NEW schema version. DO NOT EDIT the existing schema versions.** + +In general, for safety, schema version upgrades should not delete data. Schema version downgrades will simply throw an error and exit for now. + +## how to view the logs on the database server (legion.cyberia.club) + +`sudo -u postgres pg_dump capsul-flask | gzip -9 > capsul-backup-2021-02-15.gz` diff --git a/docs/deployment.md b/docs/deployment.md new file mode 100644 index 0000000..305ad2d --- /dev/null +++ b/docs/deployment.md @@ -0,0 +1,68 @@ +# Deploying Capsul on a server + +## Installing prerequisites for Spoke Mode + +On your spoke (see [Architecture](./architecture.md) You'll need `libvirtd`, `dnsmasq`, and `qemu-kvm`, plus a `/tank` diectory with some operating system images in it: + +``` +sudo apt install libvirt-daemon-system virtinst git dnsmasq qemu qemu-kvm +sudo mkdir -p /var/www /tank/{vm,img,config} +sudo mkdir -p /tank/img/debian/10 +cd !$ +sudo wget https://cloud.debian.org/images/cloud/buster/20201023-432/debian-10-genericcloud-amd64-20201023-432.qcow2 -O root.img.qcow2 +``` + +TODO: network set-up +TODO: cyberia-cloudinit.yml + +## Deploying capsul-flask + +### Extra Manual™ + +Follow the [local set-up instructions](./local-set-up.md) on your server. + +Make sure to set `BASE_URL` correctly, generate your own secret tokens, and +configure your own daemon management for the capsul-flask server (e.g. writing +init scripts, or SystemD unit files). + +Use the suggested `gunicorn` command (with appropriately-set address and port), +instead of `flask run`, to launch the server. + +TODO: cron runner + +### Using vanilla Docker Swarm + +Download the Co-op Cloud swarm `compose.yml`: + +```sh +wget https://git.autonomic.zone/coop-cloud/capsul/src/branch/main/compose.yml +``` + +Optionally, download add-on compose files for Stripe, BTCPay, and Spoke Mode: + +```sh +wget https://git.autonomic.zone/coop-cloud/capsul/src/branch/main/compose.{stripe,btcpay,spoke}.yml +``` + +Then, create a `.env` file and configure appropriately -- you probably want to +define most settings in [the Co-op Cloud `.envrc.sample` +file](https://git.autonomic.zone/coop-cloud/capsul/src/branch/main/.envrc.sample). + +Load the environment variables (using Python `direnv`, or a manual `set -a && source .env && set +a`), insert any necessary secrets, then run the deployment: + +```sh +docker stack deploy -c compose.yml -c compose.stripe.yml your_capsul +``` + +(where you'd add an extra `-c compose.btcpay.yml` for each optional compose file +you want, and set `your_capsul` to the "stack name" you want). + +TODO: cron runner + +### Using Co-op Cloud / Docker Swarm + +Follow [the guide in the README for the Co-op Cloud capsul package](https://git.autonomic.zone/coop-cloud/capsul/). + +### Using docker-compose + +TODO diff --git a/readme/btcpay_sin_pairing.jpg b/docs/images/btcpay_sin_pairing.jpg similarity index 100% rename from readme/btcpay_sin_pairing.jpg rename to docs/images/btcpay_sin_pairing.jpg diff --git a/readme/btcpayment_process.drawio b/docs/images/btcpayment_process.drawio similarity index 100% rename from readme/btcpayment_process.drawio rename to docs/images/btcpayment_process.drawio diff --git a/readme/btcpayment_process.png b/docs/images/btcpayment_process.png similarity index 100% rename from readme/btcpayment_process.png rename to docs/images/btcpayment_process.png diff --git a/readme/generate_btcpay_keys.py b/docs/images/generate_btcpay_keys.py similarity index 100% rename from readme/generate_btcpay_keys.py rename to docs/images/generate_btcpay_keys.py diff --git a/readme/hub-and-spoke.xml b/docs/images/hub-and-spoke.xml similarity index 100% rename from readme/hub-and-spoke.xml rename to docs/images/hub-and-spoke.xml diff --git a/readme/hub-and-spoke1.png b/docs/images/hub-and-spoke1.png similarity index 100% rename from readme/hub-and-spoke1.png rename to docs/images/hub-and-spoke1.png diff --git a/readme/hub-and-spoke2.png b/docs/images/hub-and-spoke2.png similarity index 100% rename from readme/hub-and-spoke2.png rename to docs/images/hub-and-spoke2.png diff --git a/readme/paired.jpg b/docs/images/paired.jpg similarity index 100% rename from readme/paired.jpg rename to docs/images/paired.jpg diff --git a/docs/local-set-up.md b/docs/local-set-up.md new file mode 100644 index 0000000..4f3adef --- /dev/null +++ b/docs/local-set-up.md @@ -0,0 +1,69 @@ +# How to run Capsul locally + +## With Docker + +If you have Docker and Docker-Compose installed, you can use the +`3wordchant/capsul-flask` Docker image to launch capsul-flask, and a Postgres +database server, for you: + +```sh +docker-compose up +``` + +docker-compose will read settings from your `.env` file; you can set any of the +options mentioned in the [configuration documentation](./configuration.md). + +## Manually + +Ensure you have the pre-requisites for the psycopg2 Postgres database adapter package: + +```sh +sudo apt install python3-dev libpq-dev +pg_config --version +``` + +Ensure you have the wonderful `pipenv` python package management and virtual environment cli: + +```sh +sudo apt install pipenv +``` + +Create python virtual environment and install packages: + +```sh +pipenv install +``` + +Run an instance of Postgres (I used docker for this, you can use whatever you want, point is its listening on `localhost:5432`): + +```sh +docker run --rm -it -e POSTGRES_PASSWORD=dev -p 5432:5432 postgres +``` + +Run the app + +```sh +pipenv run flask run +``` + +or, using Gunicorn: + +```sh +pipenv run gunicorn --bind 127.0.0.1:5000 -k gevent --worker-connections 1000 app:app +``` + +Note that by default when running locally, the `SPOKE_MODEL` is set to `mock`, meaning that it won't actually try to spawn vms. + +## Crediting your account + +Once you log in for the first time, you will want to give yourself some free capsulbux so you can create fake capsuls for testing. + +```sh +pipenv run flask cli sql -c "INSERT INTO payments (email, dollars) VALUES ('', 20.00)" +``` + +## Running scheduled tasks: + +```sh +pipenv run flask cli cron-task +```