Python Services Development

Dependecy Management

This project uses Poetry instead of pip for dependencies management. Installation instructions are here. Poetry keeps the list of dependencies in the pyproject.toml file. Each service and each library has a separate pyproject.toml file. By default, poetry creates virtual environments in {cache-dir}/virtualenvs ({cache-dir}\virtualenvs on Windows). Instead of using the default virtualenvs directory, we will instruct poetry to create .venv directory next to each pyproject.toml file, by running the command poetry config virtualenvs.in-project true. So now a .venv folder will be created automatically for each service during poetry install.

Services Development

Create .env file

Copy the .env.example as .env in the same directory. Don't forget to set the DEBUG variable to True

Installing Dependencies

Install the dependencies for each service/library you want to work. E.g. for harvester service:

cd services/harvester
poetry install

Installing a library to a service with poetry

In order to install a suite5 library to a service, you have to add it in the service's pyproject.toml dev-dependencies. E.g. if you want to add the messaging lib, you have to add messaging = {path = "../../libs/messaging", develop = true}. For more info about poetry's relative installs, visit poetry docs

Running tests

cd services/harvester
source .venv/bin/activate
pytest

Running tests with coverage

cd services/harvester
source .venv/bin/activate
coverage run --source ./ -m pytest
coverage report -m

Run service

cd services/harvester
source .venv/bin/activate
python src/main.py -c ./files/<file.json>

Libraries Development

Libraries are developed through services, given that they are installed in develop mode.

Running tests

cd services/harvester
source .venv/bin/activate
pytest

Running tests with coverage

cd services/harvester
source .venv/bin/activate
coverage run --source ./ -m pytest
coverage report -m

Using tools in Development

In tools directory, there is a docker-compose file that deploys services needed for local development. In scripts folder, there are python scripts that can be executed to ease up development (e.g. create presigned urls for retrieving or uploading files to MinIO). Tools folder is considered a separate project (that uses poetry), so it has its own dependencies that need to be installed.