forest's ReadMe docs changes

This commit is contained in:
forest 2021-07-21 13:35:30 -05:00 committed by 3wc
parent 6ee7a42686
commit bddf42b750
8 changed files with 158 additions and 46 deletions

View File

@ -1,11 +1,50 @@
# capsulflask
# capsul-flask
Python Flask web application for capsul.org
![screenshot of capsul.org home page](./docs/capsul.webp)
How about a trip to the the `docs/` folder?
- [Setting up Capsul locally](./docs/local-set-up.md)
- [Hub-and-spoke architecture](./docs/architecture.md)
- [Deplying Capsul on a server](./docs/deployment.md)
- [Configuring Capsul](./docs/configuration.md)
- [Receiving cryptocurrency payments with BTCPay](./docs/btcpay.md)
- [Working with the database](./docs/database.md)
Python Flask web application implementing user accounts, payment, and virtual machine management for a smol "virtual machine (vm) as a service" aka "cloud compute" provider. Originally developer by [Cyberia Computer Club](https://cyberia.club) for https://capsul.org
`capsul-flask` integrates with [Stripe](https://stripe.com/) as a credit card processor, and [BTCPay Server](https://github.com/btcpayserver/btcpayserver-docker) as a cryptocurrency payment processor.
`capsul-flask` invokes [shell-scripts](./capsulflask/shell_scripts/) to create/manage [libvirt/qemu](https://www.libvirt.org/manpages/virsh.html) vms, and it depends on `dnsmasq` to act as the DHCP server for the vms.
`capsul-flask` has a ["hub and spoke" architecture](./architecture.md). The "Hub" runs the web application and talks to the Postrges database, while the "Spoke"(s) are responsible for creating/managing virtual machines. In this way, capsul can be scaled to span more than one machine. One instance of the capsul-flask application can run in both hub mode and spoke mode at the same time, however there must only be one instance of the app running in "Hub" mode at any given time.
## Quickstart (run capsul-flask on your computer in development mode)
```
# get an instance of postgres running locally on port 5432
# (you don't have to use docker, but we thought this might be the easiest for a how-to example)
docker run --rm -it -e POSTGRES_PASSWORD=dev -p 5432:5432 postgres &
# install dependencies
sudo apt install pipenv python3-dev libpq-dev
# download and run
git clone https://giit.cyberia.club/~forest/capsul-flask
cd capsul-flask
pipenv install
pipenv run flask run
```
Interested in learning more? How about a trip to the the `docs/` folder:
- [**Setting up capsul-flask locally**](./docs/local-set-up.md)
- [Manually](./docs/local-set-up.md#manually)
- [With docker-compose](./docs/local-set-up.md#docker_compose)
- [**Configuring `capsul-flask`**](./docs/configuration.md)
- [Example configuration from capsul.org (production)](./docs/configuration.md#example)
- [Loading variables from files (docker secrets)](./docs/configuration.md#docker_secrets)
- [**`capsul-flask`'s relationship to its Database Server**](./docs/database.md)
- [Database schema management (schema versions)](./docs/database.md#schema_management)
- [Running manual database queries](./docs/database.md#manual_queries)
- [**`capsul-flask`'s hub-and-spoke architecture**](./docs/architecture.md)
- [**Deploying capsul-flask on a server**](./docs/deployment.md)
- [Installing prerequisites for Spoke Mode](./docs/deployment.md#spoke_mode_prerequisites)
- [Deploying capsul-flask manually](./docs/deployment.md#deploy_manually)
- [Deploying capsul-flask with coop-cloud's docker-swarm configuration](./docs/deployment.md#coop_cloud_docker)
- [Deploying capsul-flask with coop-cloud's `abra` deployment tool](./docs/deployment.md#coop_cloud_abra)
- [**Accepting cryptocurrency payments with BTCPay Server**](./docs/btcpay.md)
- [Setting up the BTCPAY_PRIVATE_KEY](./docs/btcpay.md#BTCPAY_PRIVATE_KEY)
- [Testing cryptocurrency payments](./docs/btcpay.md#testing)
- [Sequence diagram explaining how BTC payment process works (how we accept 0-confirmation transactions 😀)](./docs/btcpay.md#0_conf_diagram)

View File

@ -1,5 +1,7 @@
# hub-and-spoke architecture
The "Hub" runs the web application and talks to the Postrges database, while the "Spoke"s are responsible for creating/managing virtual machines. One instance of the capsul-flask application can run in hub mode and spoke mode at the same time.
![](images/hub-and-spoke1.png)
This diagram was created with https://app.diagrams.net/.

View File

@ -1,4 +1,4 @@
# Receiving cryptocurrency payments with BTCPay
## <a name="BTCPAY_PRIVATE_KEY"></a>Setting up the BTCPAY_PRIVATE_KEY
Generate a private key and the accompanying bitpay SIN for the btcpay API client.
@ -55,12 +55,12 @@ BTCPAY_PRIVATE_KEY='-----BEGIN EC PRIVATE KEY-----\nEXAMPLEIArx/EXAMPLEKH23EXAMP
-----
## testing cryptocurrency payments
## <a name="testing"></a>testing cryptocurrency payments
I used litecoin to test cryptocurrency payments, because its the simplest & lowest fee cryptocurrency that BTCPay server supports. You can download the easy-to-use litecoin SPV wallet `electrum-ltc` from [github.com/pooler/electrum-ltc](https://github.com/pooler/electrum-ltc) or [electrum-ltc.org](https://electrum-ltc.org/), set up a wallet, and then either purchase some litecoin from an exchange, or ask Forest for some litecoin to use for testing.
I used litecoin to test cryptocurrency payments, because its the simplest & lowest fee cryptocurrency that BTCPay server supports. You can download the easy-to-use litecoin SPV wallet `electrum-ltc` from [github.com/pooler/electrum-ltc](https://github.com/pooler/electrum-ltc) or [electrum-ltc.org](https://electrum-ltc.org/), set up a wallet, and then either purchase some litecoin from an exchange, or [ask Forest for some litecoin](https://sequentialread.com/capsul-rollin-onwards-with-a-web-application/#sqr-comment-container) to use for testing.
## sequence diagram explaining how BTC payment process works
## <a name="0_conf_diagram"></a>sequence diagram explaining how BTC payment process works (how we accept 0-confirmation transactions 😀)
![btcpayment_process](readme/btcpayment_process.png)

BIN
docs/capsul.webp Normal file

Binary file not shown.

After

Width:  |  Height:  |  Size: 10 KiB

View File

@ -6,7 +6,7 @@ Create a `.env` file to set up the application configuration:
nano .env
```
You can enter any environment variables referenced in `__init__.py` to this file.
You can enter any environment variables referenced in [`__init__.py`](../capsulflask/__init__.py) to this file.
For example you may enter your SMTP credentials like this:
```
@ -15,7 +15,55 @@ MAIL_DEFAULT_SENDER=forest@nullhex.com
MAIL_PASSWORD=**************
```
## Loading variables from files
## <a name="example"></a>Example configuration from capsul.org (production):
```
#LOG_LEVEL=DEBUG
BASE_URL="https://capsul.org"
# hub url is used by the SPOKE_MODE to contact the hub. Since this server is the hub,
# this is fine. In fact it runs into problems (routing related?) when I set it to capsul.org.
# similarly the baikal "spoke" (set up in the hosts table in the db) has "http://localhost:5000" as the https_url
HUB_URL="http://localhost:5000"
HUB_MODE_ENABLED="t"
SPOKE_MODE_ENABLED="t"
HUB_MODEL="capsul-flask"
SPOKE_MODEL="shell-scripts"
SPOKE_HOST_ID="baikal"
SPOKE_HOST_TOKEN="<redacted>"
HUB_TOKEN="<redacted>"
# smtp.. see https://flask-mail.readthedocs.io/en/latest/#configuring-flask-mail
MAIL_SERVER="smtp.nullhex.com"
# MAIL_USE_SSL means SMTP with STARTTLS
MAIL_USE_SSL=true
# MAIL_USE_TLS means SMTP wrapped in TLS
MAIL_USE_TLS=false
MAIL_PORT="465"
MAIL_USERNAME="capsul@nullhex.com"
MAIL_PASSWORD="<redacted>"
MAIL_DEFAULT_SENDER="capsul@nullhex.com"
# stripe
STRIPE_SECRET_KEY="sk_live_<redacted>"
STRIPE_PUBLISHABLE_KEY="pk_live_tGDHY7kBwqC71b4F0N7LZdGl00GZOw0iNJ"
# internal
SECRET_KEY="<redacted>"
POSTGRES_CONNECTION_PARAMETERS="sslmode=verify-full sslrootcert=letsencrypt-root-ca.crt host=postgres.cyberia.club port=5432 ...<redacted>"
# btcpay server
BTCPAY_URL="https://beeteeceepae2.cyberia.club"
BTCPAY_PRIVATE_KEY='-----BEGIN EC PRIVATE KEY-----\n<redacted>\n-----END EC PRIVATE KEY-----'
```
## <a name="docker_secrets"></a>Loading variables from files (docker secrets)
To support [Docker Secrets](https://docs.docker.com/engine/swarm/secrets/), you can also load secret values from files for example, to load `MAIL_PASSWORD` from `/run/secrets/mail_password`, set
```sh

View File

@ -1,6 +1,20 @@
# Working with the Capsul database
# capsul-flask's relationship to its Database Server
## Running manual database queries
Capsul has a ["hub and spoke" architecture](./architecture.md). The "Hub" runs the web application and talks to the Postrges database, while the "Spoke"s are responsible for creating/managing virtual machines. One instance of the capsul-flask application can run in both hub mode and spoke mode at the same time, however there must only be one instance of the app running in "Hub" mode at any given time.
The Postgres connections parameters are [configurable](./configuration.md).
## <a name="schema_management"></a>Database schema management (schema versions)
capsul-flask has a concept of a schema version. When the application starts, it will query the database for a table named `schemaversion` that has one row and one column (`version`). If the `version` it finds is not equal to the `desiredSchemaVersion` variable set in `db.py`, it will run migration scripts from the `schema_migrations` folder one by one until the `schemaversion` table shows the correct version.
For example, the script named `02_up_xyz.sql` should contain code that migrates the database from schema version 1 to schema version 2. Likewise, the script `02_down_xyz.sql` should contain code that migrates from schema version 2 back to schema version 1.
**IMPORTANT: if you need to make changes to the schema, make a NEW schema version. DO NOT EDIT the existing schema versions.**
In general, for safety, schema version upgrades should not delete data. Schema version downgrades will simply throw an error and exit for now.
## <a name="manual_queries"></a>Running manual database queries
You can manually mess around with the database like this:
@ -31,16 +45,6 @@ $ pipenv run flask cli sql -c "SELECT id, created, email, dollars, invalidated f
1, 2020-05-05T00:00:00, forest.n.johnson@gmail.com, 20.00, TRUE
```
## Database schema management
capsulflask has a concept of a schema version. When the application starts, it will query the database for a table named `schemaversion` that has one row and one column (`version`). If the `version` it finds is not equal to the `desiredSchemaVersion` variable set in `db.py`, it will run migration scripts from the `schema_migrations` folder one by one until the `schemaversion` table shows the correct version.
For example, the script named `02_up_xyz.sql` should contain code that migrates the database from schema version 1 to schema version 2. Likewise, the script `02_down_xyz.sql` should contain code that migrates from schema version 2 back to schema version 1.
**IMPORTANT: if you need to make changes to the schema, make a NEW schema version. DO NOT EDIT the existing schema versions.**
In general, for safety, schema version upgrades should not delete data. Schema version downgrades will simply throw an error and exit for now.
## how to view the logs on the database server (legion.cyberia.club)
`sudo -u postgres pg_dump capsul-flask | gzip -9 > capsul-backup-2021-02-15.gz`

View File

@ -1,6 +1,8 @@
# Deploying Capsul on a server
## Installing prerequisites for Spoke Mode
Capsul has a ["hub and spoke" architecture](./architecture.md). The "Hub" runs the web application and talks to the Postrges database, while the "Spoke"s are responsible for creating/managing virtual machines. One instance of the capsul-flask application can run in both hub mode and spoke mode at the same time, however there must only be one instance of the app running in "Hub" mode at any given time.
## <a name="spoke_mode_prerequisites"></a>Installing prerequisites for Spoke Mode
On your spoke (see [Architecture](./architecture.md) You'll need `libvirtd`, `dnsmasq`, and `qemu-kvm`, plus a `/tank` diectory with some operating system images in it:
@ -17,7 +19,7 @@ TODO: cyberia-cloudinit.yml
## Deploying capsul-flask
### Extra Manual™
### <a name="deploy_manually"></a>Manually
Follow the [local set-up instructions](./local-set-up.md) on your server.
@ -28,9 +30,26 @@ init scripts, or SystemD unit files).
Use the suggested `gunicorn` command (with appropriately-set address and port),
instead of `flask run`, to launch the server.
TODO: cron runner
For example, here is the SystemD service unit file we use in production for `capsul.org`:
### Using vanilla Docker Swarm
```
[Unit]
Description=capsul-flask virtual machines as a service
After=network.target
[Service]
ExecStart=/usr/local/bin/pipenv run gunicorn --bind 127.0.0.1:5000 -k gevent --worker-connections 1000 app:app
Restart=on-failure
WorkingDirectory=/opt/capsul-flask
[Install]
WantedBy=multi-user.target
```
TODO: cron runner is required to run maintenance tasks for now, but in the future we want to build this into the python based task scheduler.
### <a name="coop_cloud_docker"></a> Using Co-op Cloud's vanilla Docker Swarm configuration
Download the Co-op Cloud swarm `compose.yml`:
@ -59,7 +78,7 @@ you want, and set `your_capsul` to the "stack name" you want).
TODO: cron runner
### Using Co-op Cloud / Docker Swarm
### <a name="coop_cloud_abra"></a> Using Co-op Cloud's `abra` deployment tool
Follow [the guide in the README for the Co-op Cloud capsul package](https://git.autonomic.zone/coop-cloud/capsul/).

View File

@ -1,19 +1,6 @@
# How to run Capsul locally
## With Docker
If you have Docker and Docker-Compose installed, you can use the
`3wordchant/capsul-flask` Docker image to launch capsul-flask, and a Postgres
database server, for you:
```sh
docker-compose up
```
docker-compose will read settings from your `.env` file; you can set any of the
options mentioned in the [configuration documentation](./configuration.md).
## Manually
## <a name="manually"></a>Manually
Ensure you have the pre-requisites for the psycopg2 Postgres database adapter package:
@ -67,3 +54,16 @@ pipenv run flask cli sql -c "INSERT INTO payments (email, dollars) VALUES ('<you
```sh
pipenv run flask cli cron-task
```
## <a name="docker_compose"></a>Run locally with docker-compose
If you have Docker and Docker-Compose installed, you can use the
`3wordchant/capsul-flask` Docker image to launch capsul-flask, and a Postgres
database server, for you:
```sh
docker-compose up
```
docker-compose will read settings from your `.env` file; you can set any of the
options mentioned in the [configuration documentation](./configuration.md).