Add preliminary automated tests #1

Merged

3wordchant merged 16 commits from tests into master 2021-08-16 00:11:16 +00:00

Conversation 4 Commits 16 Files Changed 27 +1046 -565

3wordchant commented 2021-07-19 21:09:26 +00:00

Owner

Copy Link

As discussed in #services:cyberia.club — automated tests could make it safer to contribute code, easier to review new code, and much easier to refactor, or upgrade Python dependencies.

To run tests:

1. create a Postgres database called capsulflask_test
2. run python -m unittest

Architecture

I tried to make the absolute minimal changes to be able to override settings in tests – possible alternative approaches include accepting an argument to create_app() to define which env file to load, or adding conditional logic to create_app() to pre-load specific settings before running load_dotenv() – but allowing env vars to override dotenv vars seemed cleanest. (Thanks @forest for improving on this approach)

Creating test databases

One outstanding question is how to initialise/reinitialise the test database.

Currently, the tests rely on the existence of a capsulflask_test database on localhost, accessible by the postgres user with password dev.

I create this manually using:
docker exec -it d1702306f409 psql -U postgres createdb -O postgres capsulflask_test;

In between test runs, you can either drop and recreate that database, or manually clear data using:

docker exec -it d1702306f409 psql -U postgres capsulflask_test -c "DELETE FROM vms; DELETE FROM login_tokens; DELETE FROM ssh_public_keys; DELETE FROM api_tokens; DELETE FROM accounts;

Test coverage

This tests the "landing" (public) pages, login, capsul index and creation. I didn't add automated coverage reporting yet, unclear if that seems useful.

As discussed in #services:cyberia.club — automated tests could make it safer to contribute code, easier to review new code, and much easier to refactor, or upgrade Python dependencies. To run tests: 1. create a Postgres database called `capsulflask_test` 2. run `python -m unittest` ## Architecture I tried to make the absolute minimal changes to be able to override settings in tests – possible alternative approaches include accepting an argument to create_app() to define which env file to load, or adding conditional logic to create_app() to pre-load specific settings before running load_dotenv() – but allowing env vars to override dotenv vars seemed cleanest. (Thanks @forest for improving on this approach) ## Creating test databases One outstanding question is how to initialise/reinitialise the test database. Currently, the tests rely on the existence of a capsulflask_test database on localhost, accessible by the postgres user with password dev. I create this manually using: ` docker exec -it d1702306f409 psql -U postgres createdb -O postgres capsulflask_test;` In between test runs, you can either drop and recreate that database, or manually clear data using: `docker exec -it d1702306f409 psql -U postgres capsulflask_test -c "DELETE FROM vms; DELETE FROM login_tokens; DELETE FROM ssh_public_keys; DELETE FROM api_tokens; DELETE FROM accounts;` ## Test coverage This tests the "landing" (public) pages, login, capsul index and creation. I didn't add automated coverage reporting yet, unclear if that seems useful.

3wordchant added 4 commits 2021-07-19 21:09:27 +00:00

Basic testing using flask-testing ... 665e087bd4

This commit makes it possible to override settings during tests, by
switching capsulflask/__init__.py to a "create_app" pattern, and using
`dotenv_values` instead of `load_dotenv`.

The create_app() method returns a Flask app instance, to give
more control over when to initialise the app. This allows setting
environment variables in test files.

Then, use dotenv_values to override loaded .env variables with ones from
the environment, so that tests can set `POSTGRES_CONNECTION_PARAMETERS`
and `SPOKE_MODEL` (possibly others in future..).

Inital tests for the "landing" pages, and login / activation, are
included.

Initial console tests ... 4c7caa1a38

NB capsul create isn't working properly, see #83

Add SSH key tests 9823bda4a2

Disable VM creation check for the moment 7b9e9debab

decentral1se approved these changes 2021-07-20 21:09:08 +00:00

decentral1se left a comment

Collaborator

Copy Link

LGTM! Super nice to get some tests in.

FYI, I will note that Pytest is the de facto standard for testing which is often picked up instead of good 'ol nosetest. It has a plugin in https://pytest-flask.readthedocs.io/en/latest/ and there is a rich ecosystem of plugins in general in https://docs.pytest.org/en/latest/reference/plugin_list.html. For example, there is a plugin to help with setting up + wiping the database between tests: https://pypi.org/project/pytest-postgresql/.

LGTM! Super nice to get some tests in. FYI, I will note that Pytest is the de facto standard for testing which is often picked up instead of good 'ol nosetest. It has a plugin in https://pytest-flask.readthedocs.io/en/latest/ and there is a rich ecosystem of plugins in general in https://docs.pytest.org/en/latest/reference/plugin_list.html. For example, there is a plugin to help with setting up + wiping the database between tests: https://pypi.org/project/pytest-postgresql/.

3wordchant commented 2021-07-20 21:54:30 +00:00

Author

Owner

Copy Link

Ha, I kept flip flopping between unittest and pytest and ended up going with unittest (via flask-testing) copypastad from a previous project.

Do you think it's better to go with pytest straight from the start, if we're going to? i.e. is it worth merging this as-is or should we switch first?

Ha, I kept flip flopping between `unittest` and `pytest` and ended up going with `unittest` (via flask-testing) copypastad from a previous project. Do you think it's better to go with pytest straight from the start, if we're going to? i.e. is it worth merging this as-is or should we switch first?

decentral1se commented 2021-07-20 22:09:47 +00:00

Collaborator

Copy Link

You can get Pytest to run your Nosetests so I think you can just merge this as-is. See https://docs.pytest.org/en/6.2.x/nose.html for more.

You can get Pytest to run your Nosetests so I think you can just merge this as-is. See https://docs.pytest.org/en/6.2.x/nose.html for more.

👀 1

3wordchant force-pushed tests from 0268e4c97d to 62c7355b4c 2021-07-23 01:05:12 +00:00 Compare

3wordchant added 1 commit 2021-07-23 16:01:07 +00:00

Fix capsul create tests, post-test cleanup, tidy merge e1867eb430

forest added 1 commit 2021-07-27 17:03:07 +00:00

remove redundant get_vms() and add testing documentation from pull ... 4cf11798aa

request

forest added 4 commits 2021-07-27 18:57:26 +00:00

trying to get tests to pass with hub_model=capsulflask 8a4794a344

add debug test log 3fb8254c15

blah failed attempts at getting tests to log aa67a1e1b2

trying to do CaptureLogOutputDuringTestsFilter (but no worky yet) 56b00934be

3wordchant added 4 commits 2021-08-05 23:51:45 +00:00

breaking up after abusive relationship with logger 45d7e2c62e

getting unit tests to log properly f5640a1d01

documenting the janky tests logs situation 6d52f56d27

create TestHTTPClient that uses werkzueg test client, tests are passing c0bc65ed3d

3wordchant commented 2021-08-06 00:10:33 +00:00

Author

Owner

Copy Link

@forest do you think this is good to merge now? Or anything else we should improve about it? CI is on my mind, personally I think it would still be useful to have manual tests for the time being.

@forest do you think this is good to merge now? Or anything else we should improve about it? CI is on my mind, personally I think it would still be useful to have manual tests for the time being.

forest commented 2021-08-06 20:39:42 +00:00

Collaborator

Copy Link

I would love to be able to fix the logging properly without resorting to wrapping it like i did. But I dunno when I will do that and IDK if it should be a blocker for merging this. its the only really big problem I see with merging it right now.

I would love to be able to fix the logging properly without resorting to wrapping it like i did. But I dunno when I will do that and IDK if it should be a blocker for merging this. its the only really big problem I see with merging it right now.

3wordchant force-pushed tests from f553ab2c98 to db48cd49c1 2021-08-16 00:01:18 +00:00 Compare

3wordchant force-pushed tests from db48cd49c1 to 13b2c05476 2021-08-16 00:07:49 +00:00 Compare

3wordchant merged commit 171c3252e4 into master 2021-08-16 00:11:16 +00:00

3wordchant referenced this issue from a commit 2021-08-16 00:11:16 +00:00

Merge pull request 'Add preliminary automated tests' (#1) from tests into master

Sign in to join this conversation.

Reviewers

No reviewers

decentral1se

Labels

Clear labels

No items

No Label

Milestone

Clear milestone

No items

No Milestone

Assignees

Clear assignees

No Assignees

3 Participants

Notifications

Subscribe

Due Date

The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: 3wordchant/capsul-flask#1

No description provided.