mirsal/capsul-flask
0
0
Fork 0
forked from 3wordchant/capsul-flask
Code Issues Pull Requests Projects Releases Wiki Activity

Split README up into separate files

Browse Source
This commit is contained in:
3wc 2021-07-20 11:34:56 +02:00
parent f848eda931
commit 6ee7a42686
15 changed files with 307 additions and 223 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

230
README.md
View File

@ -2,226 +2,10 @@
Python Flask web application for capsul.org
## how to run locally
Ensure you have the pre-requisites for the psycopg2 Postgres database adapter package
```
sudo apt install python3-dev libpq-dev
pg_config --version
```
Ensure you have the wonderful `pipenv` python package management and virtual environment cli
```
sudo apt install pipenv
```
Create python virtual environment and install packages
```
# install deps
pipenv install
```
Run an instance of Postgres (I used docker for this, you can use whatever you want, point is its listening on localhost:5432)
```
docker run --rm -it -e POSTGRES_PASSWORD=dev -p 5432:5432 postgres
```
Run the app
```
pipenv run flask run
```
Run the app in gunicorn:
```
pipenv run gunicorn --bind 127.0.0.1:5000 -k gevent --worker-connections 1000 app:app
```
Once you log in for the first time, you will want to give yourself some free capsulbux so you can create fake capsuls for testing.
Note that by default when running locally, the `SPOKE_MODEL` is set to `mock`, meaning that it won't actually try to spawn vms.
```
pipenv run flask cli sql -c "INSERT INTO payments (email, dollars) VALUES ('<your email address here>', 20.00)"
```
## configuration:
Create a `.env` file to set up the application configuration:
```
nano .env
```
You can enter any environment variables referenced in `__init__.py` to this file.
For example you may enter your SMTP credentials like this:
```
MAIL_USERNAME=forest@nullhex.com
MAIL_DEFAULT_SENDER=forest@nullhex.com
MAIL_PASSWORD=**************
```
## how to view the logs on the database server (legion.cyberia.club)
`sudo -u postgres pg_dump capsul-flask | gzip -9 > capsul-backup-2021-02-15.gz`
-----
## cli
You can manually mess around with the database like this:
```
pipenv run flask cli sql -f test.sql
```
```
pipenv run flask cli sql -c 'SELECT * FROM vms'
```
This one selects the vms table with the column name header:
```
pipenv run flask cli sql -c "SELECT string_agg(column_name::text, ', ') from information_schema.columns WHERE table_name='vms'; SELECT * from vms"
```
How to modify a payment manually, like if you get a chargeback or to fix customer payment issues:
```
$ pipenv run flask cli sql -c "SELECT id, created, email, dollars, invalidated from payments"
1, 2020-05-05T00:00:00, forest.n.johnson@gmail.com, 20.00, FALSE
$ pipenv run flask cli sql -c "UPDATE payments SET invalidated = True WHERE id = 1"
1 rows affected.
$ pipenv run flask cli sql -c "SELECT id, created, email, dollars, invalidated from payments"
1, 2020-05-05T00:00:00, forest.n.johnson@gmail.com, 20.00, TRUE
```
How you would kick off the scheduled task:
```
pipenv run flask cli cron-task
```
-----
## postgres database schema management
capsulflask has a concept of a schema version. When the application starts, it will query the database for a table named
`schemaversion` that has one row and one column (`version`). If the `version` it finds is not equal to the `desiredSchemaVersion` variable set in `db.py`, it will run migration scripts from the `schema_migrations` folder one by one until the `schemaversion` table shows the correct version.
For example, the script named `02_up_xyz.sql` should contain code that migrates the database from schema version 1 to schema version 2. Likewise, the script `02_down_xyz.sql` should contain code that migrates from schema version 2 back to schema version 1.
**IMPORTANT: if you need to make changes to the schema, make a NEW schema version. DO NOT EDIT the existing schema versions.**
In general, for safety, schema version upgrades should not delete data. Schema version downgrades will simply throw an error and exit for now.
-----
## hub-and-spoke architecture
![](readme/hub-and-spoke1.png)
This diagram was created with https://app.diagrams.net/.
To edit it, download the <a download href="readme/hub-and-spoke.xml">diagram file</a> and edit it with the https://app.diagrams.net/ web application, or you may run the application from [source](https://github.com/jgraph/drawio) if you wish.
right now I have 2 types of operations, immediate mode and async.
both types of operations do assignment synchronously. so if the system cant assign the operation to one or more hosts (spokes),
or whatever the operation requires, then it will fail.
some operations tolerate partial failures, like, `capacity_avaliable` will succeed if at least one spoke succeeds.
for immediate mode requests (like `list`, `capacity_avaliable`, `destroy`), assignment and completion of the operation are the same thing.
for async ones, they can be assigned without knowing whether or not they succeeded (`create`).
![](readme/hub-and-spoke2.png)
This diagram was created with https://app.diagrams.net/.
To edit it, download the <a download href="readme/hub-and-spoke.xml">diagram file</a> and edit it with the https://app.diagrams.net/ web application, or you may run the application from [source](https://github.com/jgraph/drawio) if you wish.
if you issue a create, and it technically could go to any number of hosts, but only one host responds, it will succeed
but if you issue a create and somehow 2 hosts both think they own that task, it will fail and throw a big error. cuz it expects exactly 1 to own the create task
currently its not set up to do any polling. its not really like a queue at all. It's all immediate for the most part
-----
## how to setup btcpay server
Generate a private key and the accompanying bitpay SIN for the btcpay API client.
I used this code as an example: https://github.com/bitpay/bitpay-python/blob/master/bitpay/key_utils.py#L6
```
$ pipenv run python ./readme/generate_btcpay_keys.py
```
It should output something looking like this:
```
-----BEGIN EC PRIVATE KEY-----
EXAMPLEIArx/EXAMPLEKH23EXAMPLEsYXEXAMPLE5qdEXAMPLEcFHoAcEXAMPLEK
oUQDQgAEnWs47PT8+ihhzyvXX6/yYMAWWODluRTR2Ix6ZY7Z+MV7v0W1maJzqeqq
NQ+cpBvPDbyrDk9+Uf/sEaRCma094g==
-----END EC PRIVATE KEY-----
EXAMPLEwzAEXAMPLEEXAMPLEURD7EXAMPLE
```
In order to register the key with the btcpay server, you have to first generate a pairing token using the btcpay server interface.
This requires your btcpay server account to have access to the capsul store. Ask Cass about this.
Navigate to `Manage store: Access Tokens` at: `https://btcpay.cyberia.club/stores/<store-id>/Tokens`
![](readme/btcpay_sin_pairing.jpg)
Finally, send an http request to the btcpay server to complete the pairing:
```
curl -H "Content-Type: application/json" https://btcpay.cyberia.club/tokens -d "{'id': 'EXAMPLEwzAEXAMPLEEXAMPLEURD7EXAMPLE', 'pairingCode': 'XXXXXXX'}"
```
It should respond with a token:
```
{"data":[{"policies":[],"pairingCode":"XXXXXXX","pairingExpiration":1589473817597,"dateCreated":1589472917597,"facade":"merchant","token":"xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx","label":"capsulflask"}]}
```
And you should see the token in the btcpay server UI:
![](readme/paired.jpg)
Now simply set your `BTCPAY_PRIVATE_KEY` variable in `.env`
NOTE: make sure to use single quotes and replace the new lines with \n.
```
BTCPAY_PRIVATE_KEY='-----BEGIN EC PRIVATE KEY-----\nEXAMPLEIArx/EXAMPLEKH23EXAMPLEsYXEXAMPLE5qdEXAMPLEcFHoAcEXAMPLEK\noUQDQgAEnWs47PT8+ihhzyvXX6/yYMAWWODluRTR2Ix6ZY7Z+MV7v0W1maJzqeqq\nNQ+cpBvPDbyrDk9+Uf/sEaRCma094g==\n-----END EC PRIVATE KEY-----'
```
-----
## testing cryptocurrency payments
I used litecoin to test cryptocurrency payments, because its the simplest & lowest fee cryptocurrency that BTCPay server supports. You can download the easy-to-use litecoin SPV wallet `electrum-ltc` from [github.com/pooler/electrum-ltc](https://github.com/pooler/electrum-ltc) or [electrum-ltc.org](https://electrum-ltc.org/), set up a wallet, and then either purchase some litecoin from an exchange, or ask Forest for some litecoin to use for testing.
## sequence diagram explaining how BTC payment process works
![btcpayment_process](readme/btcpayment_process.png)
This diagram was created with https://app.diagrams.net/.
To edit it, download the <a download href="readme/btcpayment_process.drawio">diagram file</a> and edit it with the https://app.diagrams.net/ web application, or you may run the application from [source](https://github.com/jgraph/drawio) if you wish.
How about a trip to the the `docs/` folder?
- [Setting up Capsul locally](./docs/local-set-up.md)
- [Hub-and-spoke architecture](./docs/architecture.md)
- [Deplying Capsul on a server](./docs/deployment.md)
- [Configuring Capsul](./docs/configuration.md)
- [Receiving cryptocurrency payments with BTCPay](./docs/btcpay.md)
- [Working with the database](./docs/database.md)

26
docs/architecture.md Normal file
View File

@ -0,0 +1,26 @@
# hub-and-spoke architecture
![](images/hub-and-spoke1.png)
This diagram was created with https://app.diagrams.net/.
To edit it, download the <a download href="readme/hub-and-spoke.xml">diagram file</a> and edit it with the https://app.diagrams.net/ web application, or you may run the application from [source](https://github.com/jgraph/drawio) if you wish.
right now I have 2 types of operations, immediate mode and async.
both types of operations do assignment synchronously. so if the system cant assign the operation to one or more hosts (spokes),
or whatever the operation requires, then it will fail.
some operations tolerate partial failures, like, `capacity_avaliable` will succeed if at least one spoke succeeds.
for immediate mode requests (like `list`, `capacity_avaliable`, `destroy`), assignment and completion of the operation are the same thing.
for async ones, they can be assigned without knowing whether or not they succeeded (`create`).
![](images/hub-and-spoke2.png)
This diagram was created with https://app.diagrams.net/.
To edit it, download the <a download href="readme/hub-and-spoke.xml">diagram file</a> and edit it with the https://app.diagrams.net/ web application, or you may run the application from [source](https://github.com/jgraph/drawio) if you wish.
if you issue a create, and it technically could go to any number of hosts, but only one host responds, it will succeed
but if you issue a create and somehow 2 hosts both think they own that task, it will fail and throw a big error. cuz it expects exactly 1 to own the create task
currently its not set up to do any polling. its not really like a queue at all. It's all immediate for the most part

68
docs/btcpay.md Normal file
View File

@ -0,0 +1,68 @@
# Receiving cryptocurrency payments with BTCPay
Generate a private key and the accompanying bitpay SIN for the btcpay API client.
I used this code as an example: https://github.com/bitpay/bitpay-python/blob/master/bitpay/key_utils.py#L6
```
$ pipenv run python ./readme/generate_btcpay_keys.py
```
It should output something looking like this:
```
-----BEGIN EC PRIVATE KEY-----
EXAMPLEIArx/EXAMPLEKH23EXAMPLEsYXEXAMPLE5qdEXAMPLEcFHoAcEXAMPLEK
oUQDQgAEnWs47PT8+ihhzyvXX6/yYMAWWODluRTR2Ix6ZY7Z+MV7v0W1maJzqeqq
NQ+cpBvPDbyrDk9+Uf/sEaRCma094g==
-----END EC PRIVATE KEY-----
EXAMPLEwzAEXAMPLEEXAMPLEURD7EXAMPLE
```
In order to register the key with the btcpay server, you have to first generate a pairing token using the btcpay server interface.
This requires your btcpay server account to have access to the capsul store. Ask Cass about this.
Navigate to `Manage store: Access Tokens` at: `https://btcpay.cyberia.club/stores/<store-id>/Tokens`
![](readme/btcpay_sin_pairing.jpg)
Finally, send an http request to the btcpay server to complete the pairing:
```
curl -H "Content-Type: application/json" https://btcpay.cyberia.club/tokens -d "{'id': 'EXAMPLEwzAEXAMPLEEXAMPLEURD7EXAMPLE', 'pairingCode': 'XXXXXXX'}"
```
It should respond with a token:
```
{"data":[{"policies":[],"pairingCode":"XXXXXXX","pairingExpiration":1589473817597,"dateCreated":1589472917597,"facade":"merchant","token":"xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx","label":"capsulflask"}]}
```
And you should see the token in the btcpay server UI:
![](readme/paired.jpg)
Now simply set your `BTCPAY_PRIVATE_KEY` variable in `.env`
NOTE: make sure to use single quotes and replace the new lines with \n.
```
BTCPAY_PRIVATE_KEY='-----BEGIN EC PRIVATE KEY-----\nEXAMPLEIArx/EXAMPLEKH23EXAMPLEsYXEXAMPLE5qdEXAMPLEcFHoAcEXAMPLEK\noUQDQgAEnWs47PT8+ihhzyvXX6/yYMAWWODluRTR2Ix6ZY7Z+MV7v0W1maJzqeqq\nNQ+cpBvPDbyrDk9+Uf/sEaRCma094g==\n-----END EC PRIVATE KEY-----'
```
-----
## testing cryptocurrency payments
I used litecoin to test cryptocurrency payments, because its the simplest & lowest fee cryptocurrency that BTCPay server supports. You can download the easy-to-use litecoin SPV wallet `electrum-ltc` from [github.com/pooler/electrum-ltc](https://github.com/pooler/electrum-ltc) or [electrum-ltc.org](https://electrum-ltc.org/), set up a wallet, and then either purchase some litecoin from an exchange, or ask Forest for some litecoin to use for testing.
## sequence diagram explaining how BTC payment process works
![btcpayment_process](readme/btcpayment_process.png)
This diagram was created with https://app.diagrams.net/.
To edit it, download the <a download href="readme/btcpayment_process.drawio">diagram file</a> and edit it with the https://app.diagrams.net/ web application, or you may run the application from [source](https://github.com/jgraph/drawio) if you wish.

23
docs/configuration.md Normal file
View File

@ -0,0 +1,23 @@
# Configuring Capsul-Flask
Create a `.env` file to set up the application configuration:
```
nano .env
```
You can enter any environment variables referenced in `__init__.py` to this file.
For example you may enter your SMTP credentials like this:
```
MAIL_USERNAME=forest@nullhex.com
MAIL_DEFAULT_SENDER=forest@nullhex.com
MAIL_PASSWORD=**************
```
## Loading variables from files
To support [Docker Secrets](https://docs.docker.com/engine/swarm/secrets/), you can also load secret values from files – for example, to load `MAIL_PASSWORD` from `/run/secrets/mail_password`, set
```sh
MAIL_PASSWORD_FILE=/run/secrets/mail_password
```

46
docs/database.md Normal file
View File

@ -0,0 +1,46 @@
# Working with the Capsul database
## Running manual database queries
You can manually mess around with the database like this:
```
pipenv run flask cli sql -f test.sql
```
```
pipenv run flask cli sql -c 'SELECT * FROM vms'
```
This one selects the vms table with the column name header:
```
pipenv run flask cli sql -c "SELECT string_agg(column_name::text, ', ') from information_schema.columns WHERE table_name='vms'; SELECT * from vms"
```
How to modify a payment manually, like if you get a chargeback or to fix customer payment issues:
```
$ pipenv run flask cli sql -c "SELECT id, created, email, dollars, invalidated from payments"
1, 2020-05-05T00:00:00, forest.n.johnson@gmail.com, 20.00, FALSE
$ pipenv run flask cli sql -c "UPDATE payments SET invalidated = True WHERE id = 1"
1 rows affected.
$ pipenv run flask cli sql -c "SELECT id, created, email, dollars, invalidated from payments"
1, 2020-05-05T00:00:00, forest.n.johnson@gmail.com, 20.00, TRUE
```
## Database schema management
capsulflask has a concept of a schema version. When the application starts, it will query the database for a table named `schemaversion` that has one row and one column (`version`). If the `version` it finds is not equal to the `desiredSchemaVersion` variable set in `db.py`, it will run migration scripts from the `schema_migrations` folder one by one until the `schemaversion` table shows the correct version.
For example, the script named `02_up_xyz.sql` should contain code that migrates the database from schema version 1 to schema version 2. Likewise, the script `02_down_xyz.sql` should contain code that migrates from schema version 2 back to schema version 1.
**IMPORTANT: if you need to make changes to the schema, make a NEW schema version. DO NOT EDIT the existing schema versions.**
In general, for safety, schema version upgrades should not delete data. Schema version downgrades will simply throw an error and exit for now.
## how to view the logs on the database server (legion.cyberia.club)
`sudo -u postgres pg_dump capsul-flask | gzip -9 > capsul-backup-2021-02-15.gz`

68
docs/deployment.md Normal file
View File

@ -0,0 +1,68 @@
# Deploying Capsul on a server
## Installing prerequisites for Spoke Mode
On your spoke (see [Architecture](./architecture.md) You'll need `libvirtd`, `dnsmasq`, and `qemu-kvm`, plus a `/tank` diectory with some operating system images in it:
```
sudo apt install libvirt-daemon-system virtinst git dnsmasq qemu qemu-kvm
sudo mkdir -p /var/www /tank/{vm,img,config}
sudo mkdir -p /tank/img/debian/10
cd !$
sudo wget https://cloud.debian.org/images/cloud/buster/20201023-432/debian-10-genericcloud-amd64-20201023-432.qcow2 -O root.img.qcow2
```
TODO: network set-up
TODO: cyberia-cloudinit.yml
## Deploying capsul-flask
### Extra Manual™
Follow the [local set-up instructions](./local-set-up.md) on your server.
Make sure to set `BASE_URL` correctly, generate your own secret tokens, and
configure your own daemon management for the capsul-flask server (e.g. writing
init scripts, or SystemD unit files).
Use the suggested `gunicorn` command (with appropriately-set address and port),
instead of `flask run`, to launch the server.
TODO: cron runner
### Using vanilla Docker Swarm
Download the Co-op Cloud swarm `compose.yml`:
```sh
wget https://git.autonomic.zone/coop-cloud/capsul/src/branch/main/compose.yml
```
Optionally, download add-on compose files for Stripe, BTCPay, and Spoke Mode:
```sh
wget https://git.autonomic.zone/coop-cloud/capsul/src/branch/main/compose.{stripe,btcpay,spoke}.yml
```
Then, create a `.env` file and configure appropriately -- you probably want to
define most settings in [the Co-op Cloud `.envrc.sample`
file](https://git.autonomic.zone/coop-cloud/capsul/src/branch/main/.envrc.sample).
Load the environment variables (using Python `direnv`, or a manual `set -a && source .env && set +a`), insert any necessary secrets, then run the deployment:
```sh
docker stack deploy -c compose.yml -c compose.stripe.yml your_capsul
```
(where you'd add an extra `-c compose.btcpay.yml` for each optional compose file
you want, and set `your_capsul` to the "stack name" you want).
TODO: cron runner
### Using Co-op Cloud / Docker Swarm
Follow [the guide in the README for the Co-op Cloud capsul package](https://git.autonomic.zone/coop-cloud/capsul/).
### Using docker-compose
TODO

0
readme/btcpay_sin_pairing.jpg → docs/images/btcpay_sin_pairing.jpg
View File

Side by Side Swipe Overlay

Before

Width:  |  Height:  |  Size: 35 KiB

After

Width:  |  Height:  |  Size: 35 KiB

0
readme/btcpayment_process.drawio → docs/images/btcpayment_process.drawio
View File

0
readme/btcpayment_process.png → docs/images/btcpayment_process.png
View File

Side by Side Swipe Overlay

Before

Width:  |  Height:  |  Size: 190 KiB

After

Width:  |  Height:  |  Size: 190 KiB

0
readme/generate_btcpay_keys.py → docs/images/generate_btcpay_keys.py
View File

0
readme/hub-and-spoke.xml → docs/images/hub-and-spoke.xml
View File

0
readme/hub-and-spoke1.png → docs/images/hub-and-spoke1.png
View File

Side by Side Swipe Overlay

Before

Width:  |  Height:  |  Size: 41 KiB

After

Width:  |  Height:  |  Size: 41 KiB

0
readme/hub-and-spoke2.png → docs/images/hub-and-spoke2.png
View File

Side by Side Swipe Overlay

Before

Width:  |  Height:  |  Size: 49 KiB

After

Width:  |  Height:  |  Size: 49 KiB

0
readme/paired.jpg → docs/images/paired.jpg
View File

Side by Side Swipe Overlay

Before

Width:  |  Height:  |  Size: 22 KiB

After

Width:  |  Height:  |  Size: 22 KiB

69
docs/local-set-up.md Normal file
View File

@ -0,0 +1,69 @@
# How to run Capsul locally
## With Docker
If you have Docker and Docker-Compose installed, you can use the
`3wordchant/capsul-flask` Docker image to launch capsul-flask, and a Postgres
database server, for you:
```sh
docker-compose up
```
docker-compose will read settings from your `.env` file; you can set any of the
options mentioned in the [configuration documentation](./configuration.md).
## Manually
Ensure you have the pre-requisites for the psycopg2 Postgres database adapter package:
```sh
sudo apt install python3-dev libpq-dev
pg_config --version
```
Ensure you have the wonderful `pipenv` python package management and virtual environment cli:
```sh
sudo apt install pipenv
```
Create python virtual environment and install packages:
```sh
pipenv install
```
Run an instance of Postgres (I used docker for this, you can use whatever you want, point is its listening on `localhost:5432`):
```sh
docker run --rm -it -e POSTGRES_PASSWORD=dev -p 5432:5432 postgres
```
Run the app
```sh
pipenv run flask run
```
or, using Gunicorn:
```sh
pipenv run gunicorn --bind 127.0.0.1:5000 -k gevent --worker-connections 1000 app:app
```
Note that by default when running locally, the `SPOKE_MODEL` is set to `mock`, meaning that it won't actually try to spawn vms.
## Crediting your account
Once you log in for the first time, you will want to give yourself some free capsulbux so you can create fake capsuls for testing.
```sh
pipenv run flask cli sql -c "INSERT INTO payments (email, dollars) VALUES ('<your email address here>', 20.00)"
```
## Running scheduled tasks:
```sh
pipenv run flask cli cron-task
```