Loader
Introduction
The Loader service is responsible for storing the data to the user selected database or storage. In order to run, it needs a json configuration file as an input. Loader service is implemented in python.
Requirements
A list of services that need to be deployed (running), in order for the Loader to be fully functional:
Supported Storage Types
- MongoDB
- Elasticsearch
- File storage to MinIO
Functionality
MongoDB
Firstly, the MongoDB Loader has to retrieve the file from MinIO and "unflatten" the data, in order to be properly stored to MongoDB. It generates the metadata and stores the data to MongoDB.
File Storage
The File Storage loader retrieves the files from MinIO, generates metadata and stores back the files to MinIO (/load path).
Configuration File
A JSON configuration file is required from the Loader. Depending on the user selected storage, the configurations files will differ. The requirements of the configurations file are described in the loader_schema.py file that exists in the Loader Service.