Asset

The asset library is responsible for asset management and indexing of assets to Elasticsearch. An asset has an asset type (the default is dataset). Through this library the user can create, retrieve, update and delete an asset and can also define new asset types.

Asset Statuses

• Incomplete: The asset's metadata are not completed.
• Uploading: The asset's metadata are completed but its data are still uploading.
• Available: The asset's metadata are completed and its data are available.
• Deprecated: The asset has been deprecated.

Asset Metadata

• General Information: General information about the profile of the specific data asset (title, description, tags, reference)
• Distribution Details: Information regarding the availability and access to the specific data asset (volume, type, format, velocity, accessibility, accrual method, accrual periodicity, language)
• Extent Details: Information regarding the coverage and granularity of the data asset from a temporal and spatial perspective (temporal coverage, spatial coverage, temporal resolution, spatial resolution)
• Licensing Information: Information regarding the license and its associated terms under which a data asset is made available (access level, copyright owner, license, link, derivation, attribution, reproduction, distribution, share alike, re-context, offline retention, target purpose)
• Pricing Details: Information regarding the payment and acquisition of the specific data asset (payment method, calculation scheme, cost, currency)
• Access Policy: Information regarding who has and doesn't have access to the specific data asset

Asset Management

User must be authenticated to create new assets, have access to his/her own assets, update or delete them.

When an asset is created, the current user is set as asset's owner, its status is set as Incomplete and asset is indexed to Elasticsearch.

Only the asset's owner can update it and if any other user tries to update it then a Forbidden Exception is thrown. If asset has Incomplete status and has been already modified before (which means that its data are loaded to the storage), then its status becomes Available and the availableAt date is set as the current date. If asset has Incomplete status but has not been modified yet, then its status becomes Uploading. If temporalCoverage exists in asset's extent metadata, then its minimum and maximum values are set and if spatialCoverage exists in asset's extent metadata, then its spatial values are set. Finally, asset and its access policies are updated and the indexed asset is updated as well.

Only the asset's owner can delete it, although an asset does not really get deleted but its status becomes Deprecated instead and the indexed asset's status is updated as well.

Asset's other files are uploaded in MinIO in a bucket with name org{user.organisationId} (if user has an organisation) or user{user.id} (otherwise), where user is the asset's owner and under the path assets/asset{asset.id}/.

Indexing of assets to Elasticsearch

All the asset's fields are indexed to Elasticsearch except sample, policies and processingRules. Also, an extra field called schema is indexed as well and contains the asset's column uids, names, types and any extra values needed (minimum, maximum, spatial values, etc).

Data Loaded

When asset's data are loaded to the storage by the loader service, a Completed feedback is sent to the backend through RabbitMQ. When this feedback is received, if its message contains metadata, then the Update and index asset process follows and if its message contains filesMetadata then the Update other files' metadata process follows.

Update and index asset process: If asset's status is Uploading, then it becomes Available and the availableAt date is set as the current date. Then, the asset's volume is updated and the modifiedAt date is set as well. After that, the indexed asset's schema is updated. If temporalCoverage exists in asset's extent metadata, then its minimum and maximum values are set and if spatialCoverage exists in asset's extent metadata, then its spatial values are set. Finally, the indexed asset is updated as well.

Update other files' metadata process: If files in filesMetadata have extensions then the file formats are set and if files in filesMetadata have mime types then the file types are set. File formats and file types are saved in asset's distribution metadata. If asset's status is Uploading, then it becomes Available and the availableAt date is set as the current date. Then, the asset's metadata are updated and the modifiedAt date is set as well. Finally, the indexed asset is updated.

Available Services

getAssetTypes

Retrieves all asset types.

getAssetType

Retrieves a specific asset type by its id.

Param	Type	Description
id	number	The asset type's id

getAssetTypeByName

Retrieves a specific asset type by its name.

Param	Type	Description
name	string	The asset type's name

getAllAssets

Retrieves all assets.

getAssets

Retrieves all assets that belong to specific users and have a specific asset type (default is dataset).

Param	Type	Description
ids	number[]	The unique users' ids
assetTypeId?	number = 1	The asset type's id

getAvailableAssets

Retrieves all assets that belong to specific users, have a specific asset type (default is dataset) and have Available status.

Same parameters as the getAssets service.

getAsset

Retrieves a specific asset by its id and fetches its retrievalQueries if fetchQueries is set to true.

Param	Type	Description
id	number	The asset's id
fetchQueries?	boolean = false	The asset type's id
manager?	any	Optional transaction manager

createAsset

Creates an asset.

Param	Type	Description
assetData	CreateAssetDTO	Asset data

updateAsset

Updates an asset using its id.

Param	Type	Description
id	number	The asset's id
assetData	UpdateAssetDTO	Asset data

updateProcessingRules

Updates asset's processing rules using its id by adding them under a specific key.

Param	Type	Description
id	number	The asset's id
processingRules	any	The asset's processing rules
key	string	The processing rules' key

updateStatus

Updates asset's status using its id.

Param	Type	Description
id	number	The asset's id
status	StatusCode	The asset's status

updateTemporalSpatialMetadata

Retrieves indexed asset's temporal and spatial coverage metadata.

Param	Type	Description
assetData	UpdateAssetMetadataDTO	Asset data
indexedAsset	IndexedAsset	The indexed asset

newDataLoaded

Updates asset's volume and modifiedAt date and if asset's status is Uploading it also sets its status to Available and updates availableAt date.

Param	Type	Description
id	number	The asset's id
data	UpdateAssetMetadataDTO	Asset data

newOtherFilesDataLoaded

Updates asset's metadata and modifiedAt date and if asset's status is Uploading it also sets its status to Available and updates availableAt date.

Param	Type	Description
id	number	The asset's id
data	UpdateAssetOtherFilesMetadataDTO	Asset data

updateAndIndexAsset

Updates asset and indexes updated asset to Elasticsearch.

Param	Type	Description
id	number	The asset's id
loaderMetadata	any	Loader metadata

updateAssetVolume

Updates asset's volume.

Param	Type	Description
id	number	The asset's id
volume	number	The asset's volume

updateOtherFilesMetadata

Updates other files asset metadata and indexes updated asset to Elasticsearch.

Same parameters as the updateAndIndexAsset service.

deprecateAsset

Updates asset's status to Deprecated using its id and emits Deprecated Asset Event.

Param	Type	Description
id	number	The asset's id

assetPolicyChange

Emits Policy Changed Asset Event.

Param	Type	Description
asset	Asset	The asset

createIndexedAsset

Creates the indexed asset that is going to be indexed to Elasticsearch.

Same parameters as the assetPolicyChange service.

mergeUserMetadata

Merges user defined metadata (asset) into an existing document (indexedAsset).

Param	Type	Description
indexedAsset	IndexedAsset	The indexed asset
asset	Asset	The asset

retrieve

Retrieves an indexed asset by its id.

Param	Type	Description
id	number	The indexed asset's id

create

Creates an indexed asset.

Same parameters as the assetPolicyChange service.

update

Updates an indexed asset using its id.

Param	Type	Description
updatedAsset	IndexedAsset	The indexed asset

updateMetadata

Updates indexed asset's metadata.

Same parameters as the assetPolicyChange service.

updateSchema

Updates indexed asset's schema.

Param	Type	Description
asset	Asset	The asset
extractedMetadata	any	Extracted metadata