forest's ReadMe docs changes

README.md

@@ -1,11 +1,50 @@
-# capsulflask
+# capsul-flask
 
-Python Flask web application for capsul.org
+![screenshot of capsul.org home page](./docs/capsul.webp)
 
-How about a trip to the the `docs/` folder?
-- [Setting up Capsul locally](./docs/local-set-up.md)
-- [Hub-and-spoke architecture](./docs/architecture.md)
-- [Deplying Capsul on a server](./docs/deployment.md)
-- [Configuring Capsul](./docs/configuration.md)
-- [Receiving cryptocurrency payments with BTCPay](./docs/btcpay.md)
-- [Working with the database](./docs/database.md)
+Python Flask web application implementing user accounts, payment, and virtual machine management for a smol "virtual machine (vm) as a service" aka "cloud compute" provider.  Originally developer by [Cyberia Computer Club](https://cyberia.club) for https://capsul.org
+
+`capsul-flask` integrates with [Stripe](https://stripe.com/) as a credit card processor, and [BTCPay Server](https://github.com/btcpayserver/btcpayserver-docker) as a cryptocurrency payment processor.
+
+`capsul-flask` invokes [shell-scripts](./capsulflask/shell_scripts/) to create/manage [libvirt/qemu](https://www.libvirt.org/manpages/virsh.html) vms, and it depends on `dnsmasq` to act as the DHCP server for the vms.
+
+`capsul-flask` has a ["hub and spoke" architecture](./architecture.md). The "Hub" runs the web application and talks to the Postrges database, while the "Spoke"(s) are responsible for creating/managing virtual machines. In this way, capsul can be scaled to span more than one machine. One instance of the capsul-flask application can run in both hub mode and spoke mode at the same time, however there must only be one instance of the app running in "Hub" mode at any given time.
+
+## Quickstart (run capsul-flask on your computer in development mode)
+
+```
+# get an instance of postgres running locally on port 5432
+# (you don't have to use docker, but we thought this might be the easiest for a how-to example)
+docker run --rm -it -e POSTGRES_PASSWORD=dev -p 5432:5432 postgres &
+
+# install dependencies 
+sudo apt install pipenv python3-dev libpq-dev
+
+# download and run
+git clone https://giit.cyberia.club/~forest/capsul-flask
+cd capsul-flask
+pipenv install
+pipenv run flask run
+```
+
+Interested in learning more? How about a trip to the the `docs/` folder:
+
+- [**Setting up capsul-flask locally**](./docs/local-set-up.md)
+  - [Manually](./docs/local-set-up.md#manually)
+  - [With docker-compose](./docs/local-set-up.md#docker_compose)
+- [**Configuring `capsul-flask`**](./docs/configuration.md)
+  - [Example configuration from capsul.org (production)](./docs/configuration.md#example)
+  - [Loading variables from files (docker secrets)](./docs/configuration.md#docker_secrets)
+- [**`capsul-flask`'s relationship to its Database Server**](./docs/database.md)
+  - [Database schema management (schema versions)](./docs/database.md#schema_management)
+  - [Running manual database queries](./docs/database.md#manual_queries)
+- [**`capsul-flask`'s hub-and-spoke architecture**](./docs/architecture.md)
+- [**Deploying capsul-flask on a server**](./docs/deployment.md)
+  - [Installing prerequisites for Spoke Mode](./docs/deployment.md#spoke_mode_prerequisites)
+  - [Deploying capsul-flask manually](./docs/deployment.md#deploy_manually)
+  - [Deploying capsul-flask with coop-cloud's docker-swarm configuration](./docs/deployment.md#coop_cloud_docker)
+  - [Deploying capsul-flask with coop-cloud's `abra` deployment tool](./docs/deployment.md#coop_cloud_abra)
+- [**Accepting cryptocurrency payments with BTCPay Server**](./docs/btcpay.md)
+  - [Setting up the BTCPAY_PRIVATE_KEY](./docs/btcpay.md#BTCPAY_PRIVATE_KEY)
+  - [Testing cryptocurrency payments](./docs/btcpay.md#testing)
+  - [Sequence diagram explaining how BTC payment process works (how we accept 0-confirmation transactions 😀)](./docs/btcpay.md#0_conf_diagram)

docs/architecture.md

@@ -1,5 +1,7 @@
 # hub-and-spoke architecture
 
+The "Hub" runs the web application and talks to the Postrges database, while the "Spoke"s are responsible for creating/managing virtual machines. One instance of the capsul-flask application can run in hub mode and spoke mode at the same time. 
+
 ![](images/hub-and-spoke1.png)
 
 This diagram was created with https://app.diagrams.net/.

docs/btcpay.md

@@ -1,4 +1,4 @@
-# Receiving cryptocurrency payments with BTCPay
+## <a name="BTCPAY_PRIVATE_KEY"></a>Setting up the BTCPAY_PRIVATE_KEY 
 
 Generate a private key and the accompanying bitpay SIN for the btcpay API client. 
 
@@ -55,12 +55,12 @@ BTCPAY_PRIVATE_KEY='-----BEGIN EC PRIVATE KEY-----\nEXAMPLEIArx/EXAMPLEKH23EXAMP
 
 -----
 
-## testing cryptocurrency payments
+## <a name="testing"></a>testing cryptocurrency payments
 
-I used litecoin to test cryptocurrency payments, because its the simplest & lowest fee cryptocurrency that BTCPay server supports. You can download the easy-to-use litecoin SPV wallet `electrum-ltc` from [github.com/pooler/electrum-ltc](https://github.com/pooler/electrum-ltc) or [electrum-ltc.org](https://electrum-ltc.org/), set up a wallet, and then either purchase some litecoin from an exchange, or ask Forest for some litecoin to use for testing.
+I used litecoin to test cryptocurrency payments, because its the simplest & lowest fee cryptocurrency that BTCPay server supports. You can download the easy-to-use litecoin SPV wallet `electrum-ltc` from [github.com/pooler/electrum-ltc](https://github.com/pooler/electrum-ltc) or [electrum-ltc.org](https://electrum-ltc.org/), set up a wallet, and then either purchase some litecoin from an exchange, or [ask Forest for some litecoin](https://sequentialread.com/capsul-rollin-onwards-with-a-web-application/#sqr-comment-container) to use for testing.
 
 
-## sequence diagram explaining how BTC payment process works
+## <a name="0_conf_diagram"></a>sequence diagram explaining how BTC payment process works (how we accept 0-confirmation transactions 😀)
 
 ![btcpayment_process](readme/btcpayment_process.png)
 

docs/configuration.md

@@ -6,7 +6,7 @@ Create a `.env` file to set up the application configuration:
 nano .env
 ```
 
-You can enter any environment variables referenced in `__init__.py` to this file.
+You can enter any environment variables referenced in [`__init__.py`](../capsulflask/__init__.py) to this file.
 
 For example you may enter your SMTP credentials like this:
 ```
@@ -15,7 +15,55 @@ MAIL_DEFAULT_SENDER=forest@nullhex.com
 MAIL_PASSWORD=**************
 ```
 
-## Loading variables from files
+## <a name="example"></a>Example configuration from capsul.org (production):
+
+```
+#LOG_LEVEL=DEBUG
+
+BASE_URL="https://capsul.org"
+
+# hub url is used by the SPOKE_MODE to contact the hub. Since this server is the hub,
+# this is fine. In fact it runs into problems (routing related?) when I set it to capsul.org.
+# similarly the baikal "spoke" (set up in the hosts table in the db) has "http://localhost:5000" as the https_url
+HUB_URL="http://localhost:5000"
+
+HUB_MODE_ENABLED="t"
+SPOKE_MODE_ENABLED="t"
+HUB_MODEL="capsul-flask"
+SPOKE_MODEL="shell-scripts"
+SPOKE_HOST_ID="baikal"
+SPOKE_HOST_TOKEN="<redacted>"
+HUB_TOKEN="<redacted>"
+
+# smtp.. see https://flask-mail.readthedocs.io/en/latest/#configuring-flask-mail
+MAIL_SERVER="smtp.nullhex.com"
+
+# MAIL_USE_SSL means SMTP with STARTTLS
+MAIL_USE_SSL=true
+
+# MAIL_USE_TLS means SMTP wrapped in TLS
+MAIL_USE_TLS=false
+
+MAIL_PORT="465"
+MAIL_USERNAME="capsul@nullhex.com"
+MAIL_PASSWORD="<redacted>"
+MAIL_DEFAULT_SENDER="capsul@nullhex.com"
+
+# stripe
+STRIPE_SECRET_KEY="sk_live_<redacted>"
+STRIPE_PUBLISHABLE_KEY="pk_live_tGDHY7kBwqC71b4F0N7LZdGl00GZOw0iNJ"
+
+# internal
+SECRET_KEY="<redacted>"
+POSTGRES_CONNECTION_PARAMETERS="sslmode=verify-full sslrootcert=letsencrypt-root-ca.crt host=postgres.cyberia.club port=5432 ...<redacted>"
+
+
+# btcpay server
+BTCPAY_URL="https://beeteeceepae2.cyberia.club"
+BTCPAY_PRIVATE_KEY='-----BEGIN EC PRIVATE KEY-----\n<redacted>\n-----END EC PRIVATE KEY-----'
+```
+
+## <a name="docker_secrets"></a>Loading variables from files (docker secrets)
 
 To support [Docker Secrets](https://docs.docker.com/engine/swarm/secrets/), you can also load secret values from files – for example, to load `MAIL_PASSWORD` from `/run/secrets/mail_password`, set
 ```sh

docs/database.md

@@ -1,6 +1,20 @@
-# Working with the Capsul database
+# capsul-flask's relationship to its Database Server
 
-## Running manual database queries
+Capsul has a ["hub and spoke" architecture](./architecture.md). The "Hub" runs the web application and talks to the Postrges database, while the "Spoke"s are responsible for creating/managing virtual machines. One instance of the capsul-flask application can run in both hub mode and spoke mode at the same time, however there must only be one instance of the app running in "Hub" mode at any given time.
+
+The Postgres connections parameters are [configurable](./configuration.md).
+
+## <a name="schema_management"></a>Database schema management (schema versions)
+
+capsul-flask has a concept of a schema version. When the application starts, it will query the database for a table named `schemaversion` that has one row and one column (`version`). If the `version` it finds is not equal to the `desiredSchemaVersion` variable set in `db.py`, it will run  migration scripts from the `schema_migrations` folder one by one until the `schemaversion` table shows the correct version.
+
+For example, the script named `02_up_xyz.sql` should contain code that migrates the database from schema version 1 to schema version 2. Likewise, the script `02_down_xyz.sql` should contain code that migrates from schema version 2 back to schema version 1.
+
+**IMPORTANT: if you need to make changes to the schema, make a NEW schema version. DO NOT EDIT the existing schema versions.**
+
+In general, for safety, schema version upgrades should not delete data. Schema version downgrades will simply throw an error and exit for now. 
+
+## <a name="manual_queries"></a>Running manual database queries
 
 You can manually mess around with the database like this:
 
@@ -31,16 +45,6 @@ $ pipenv run flask cli sql -c "SELECT id, created, email, dollars, invalidated f
 1, 2020-05-05T00:00:00, forest.n.johnson@gmail.com, 20.00, TRUE
 ```
 
-## Database schema management
-
-capsulflask has a concept of a schema version. When the application starts, it will query the database for a table named `schemaversion` that has one row and one column (`version`). If the `version` it finds is not equal to the `desiredSchemaVersion` variable set in `db.py`, it will run  migration scripts from the `schema_migrations` folder one by one until the `schemaversion` table shows the correct version.
-
-For example, the script named `02_up_xyz.sql` should contain code that migrates the database from schema version 1 to schema version 2. Likewise, the script `02_down_xyz.sql` should contain code that migrates from schema version 2 back to schema version 1.
-
-**IMPORTANT: if you need to make changes to the schema, make a NEW schema version. DO NOT EDIT the existing schema versions.**
-
-In general, for safety, schema version upgrades should not delete data. Schema version downgrades will simply throw an error and exit for now. 
-
 ## how to view the logs on the database server (legion.cyberia.club)
 
 `sudo -u postgres pg_dump capsul-flask | gzip -9 > capsul-backup-2021-02-15.gz`

docs/deployment.md

@@ -1,6 +1,8 @@
 # Deploying Capsul on a server
 
-## Installing prerequisites for Spoke Mode
+Capsul has a ["hub and spoke" architecture](./architecture.md). The "Hub" runs the web application and talks to the Postrges database, while the "Spoke"s are responsible for creating/managing virtual machines. One instance of the capsul-flask application can run in both hub mode and spoke mode at the same time, however there must only be one instance of the app running in "Hub" mode at any given time.
+
+## <a name="spoke_mode_prerequisites"></a>Installing prerequisites for Spoke Mode
 
 On your spoke (see [Architecture](./architecture.md) You'll need `libvirtd`, `dnsmasq`, and `qemu-kvm`, plus a `/tank` diectory with some operating system images in it:
 
@@ -17,7 +19,7 @@ TODO: cyberia-cloudinit.yml
 
 ## Deploying capsul-flask
 
-### Extra Manual™
+### <a name="deploy_manually"></a>Manually
 
 Follow the [local set-up instructions](./local-set-up.md) on your server. 
 
@@ -28,9 +30,26 @@ init scripts, or SystemD unit files).
 Use the suggested `gunicorn` command (with appropriately-set address and port),
 instead of `flask run`, to launch the server.
 
-TODO: cron runner
+For example, here is the SystemD service unit file we use in production for `capsul.org`:
 
-### Using vanilla Docker Swarm
+```
+[Unit]
+Description=capsul-flask virtual machines as a service
+After=network.target
+
+[Service]
+ExecStart=/usr/local/bin/pipenv run gunicorn --bind 127.0.0.1:5000 -k gevent --worker-connections 1000 app:app
+Restart=on-failure
+WorkingDirectory=/opt/capsul-flask
+
+[Install]
+WantedBy=multi-user.target
+
+```
+
+TODO: cron runner is required to run maintenance tasks for now, but in the future we want to build this into the python based task scheduler. 
+
+### <a name="coop_cloud_docker"></a> Using Co-op Cloud's vanilla Docker Swarm configuration
 
 Download the Co-op Cloud swarm `compose.yml`:
 
@@ -59,7 +78,7 @@ you want, and set `your_capsul` to the "stack name" you want).
 
 TODO: cron runner
 
-### Using Co-op Cloud / Docker Swarm
+### <a name="coop_cloud_abra"></a> Using Co-op Cloud's `abra` deployment tool
 
 Follow [the guide in the README for the Co-op Cloud capsul package](https://git.autonomic.zone/coop-cloud/capsul/).
 

docs/local-set-up.md

@@ -1,19 +1,6 @@
 # How to run Capsul locally
 
-## With Docker
-
-If you have Docker and Docker-Compose installed, you can use the
-`3wordchant/capsul-flask` Docker image to launch capsul-flask, and a Postgres
-database server, for you:
-
-```sh
-docker-compose up
-```
-
-docker-compose will read settings from your `.env` file; you can set any of the
-options mentioned in the [configuration documentation](./configuration.md).
-
-## Manually
+## <a name="manually"></a>Manually
 
 Ensure you have the pre-requisites for the psycopg2 Postgres database adapter package:
 
@@ -67,3 +54,16 @@ pipenv run flask cli sql -c "INSERT INTO payments (email, dollars) VALUES ('<you
 ```sh
 pipenv run flask cli cron-task
 ```
+
+## <a name="docker_compose"></a>Run locally with docker-compose
+
+If you have Docker and Docker-Compose installed, you can use the
+`3wordchant/capsul-flask` Docker image to launch capsul-flask, and a Postgres
+database server, for you:
+
+```sh
+docker-compose up
+```
+
+docker-compose will read settings from your `.env` file; you can set any of the
+options mentioned in the [configuration documentation](./configuration.md).