Data Model

The functionality of the data model library is only available to any user with the model-moderator role assigned. It allows the model moderator to create, edit and deprecate data models, while aiming to preserve the model consistency and validity and also inform the predictor service of changes. The UI is available via the "Models" navigation menu item (again only available to model moderators).

Model Version

Models and their concepts hold a version in order to distinguish the differences of the model through time. In general we use Semantic Versioning to manage versions. A version number is made out of 3 integers: X.Y.Z.

• X - Is the major version number. This changes only on major changes (on a stable model) which in our case means the deprecation of a highlevel concept or the import of a highlevel concept from a different data model. Change in this number is normally accompanied by cloning the model and placing the new clone with the higher version under revision (which we will look at further down)
• Y - Is the minor version number. This changes on the creation of a concept or deprecation of a field concept.
• Z - Is the patch version number. This changes on the update of any concept.

General Rules

• Two models cannot have the same name
• Two highlevel concepts of the same model cannot have the same name
• Two fields under the same highlevel concept cannot have the same name
• The name of a highlevel concept or a field must include only characters and numbers, not special characters
• After a model has been published, from draft, it cannot be fully deleted
• Two models cannot have the same domainUid, uid and majorVersion

Notes:

1. A model could have a high-level concept/ field which has the same uid as a high-level concept/ field from another model (e.g. by importing that high-level concept from the other model in this current model). In this case, in order to differentiate them we have added the domainUid column in order to know in which model each high-level concept/ field exists. The domain uid is the uid of the root concept i.e. the model concept.
2. uid: This is used to link the same concept along the different versions

Model Status and rules

Models and their concepts hold a status which signifies what kind of operations are permitted and whether the model is available for use (in mapping for example) or not. There are several rules dictating what can and cannot be done in each status and we will look at each one below.

Draft

This is the initial status a model receives when created. At this moment the model receives version 0.0.1 and any changes to the model will not change this version in any way.

Is model available for use? No

Available model actions

• Edit - Takes user to Model Manager page to allow editing
• Publish/Release - Publishes model to stable status, changes version to next major version (1.0.0) and communicates the entire new model to the predictor service
• Clone - Clones the model into another draft model (as an entirely separated model) with the word "Clone" at the end of its name
• Delete - Deletes the model completely from the database

What can the user do

• Create any highlevel concept or field
• Update any attribute of highlevel concept or field
• Rename model and change description and standards
• Delete any concept in the model, without effecting the versioning or status of the model in any way. Deleting means entirely removing the concept

Stable

This is the status after draft. In this state, the predictor already knows about this model and the version is above (or equal to) 1.0.0. Major changes to a stable model are cause for concern and stability needs to be maintained.

Is model available for use? Yes!

Available model actions

• Edit - Takes user to Model Manager page to allow editing
• Clone - Clones the model into another draft model (as an entirely separated model) with the word "Clone" at the end of its name
• Deprecate - Marks the entire model as deprecated
• Export - Exports the model in a .json format (only for users with admin role)

What can the user do

• Create any highlevel concept or field
• Update description, standards mapping, related concepts, related terms and metadata. On update, the version of the concept and its branch (ancestors and descendants) are changed. Upon update, we also inform the predictor, if there are any changes.
• Change model description and standards. Upon update, we also inform the predictor, if there are any changes.
• Deprecate a field. This will change the version of the concept and its branch (ancestors and descendants). The predictor is also informed of the deprecation.
• Deprecating a highlevel concept is a major action when the model has stable status. This will cause the model to be cloned into a new major version, where this highlevel concept will be deprecated. The previous version of the model and its concept will receive a next version id, associating the old concepts with the new version counterparts. The new major version will be in status In Review. Even though Under Review does not exist as a status, the old version will appear to be in this status because it has status Stable but has a next version id. This state will stay until either the in review model becomes either published or deleted. Any suggestions associated with the old model are moved to the new version.
• Importing a highlevel concept from another data model is a major action when the model has stable status. If fields of the imported highlevel concept are related to other highlevel concepts, those highlevel concepts will also be imported and so on. This will cause the model to be cloned into a new major version. The previous version of the model and its concept will receive a next version id, associating the old concepts with the new version counterparts. The new major version will be in status In Review. Even though Under Review does not exist as a status, the old version will appear to be in this status because it has status Stable but has a next version id. This state will stay until either the in review model becomes either published or deleted. Any suggestions associated with the old model are moved to the new version.

Under Revision

Even though this is not a real status it appears so on the screen.

Is model available for use? Yes!

Available model actions

• View - Takes user to a read-only Model Manager page
• Clone - Clones the model into another draft model (as an entirely separated model) with the word "Clone" at the end of its name

What can the user do

• Just view the model

In Review

In this status the user can make any number of changes (similar to what he/ she would be permitted to do in stable status and some actions which were allowed in draft status). Any changes to the model will not change the version of the model any further.

Is model available for use? No

Available model actions

• Edit - Takes user to Model Manager page to allow editing
• Publish/Release - Publishes the model with stable status, changes its version to the next major version and communicates all creations/ deprecations/ deletions/ updates to model manager. Moreover, the previous version of the model is deprecated
• Clone - Clones the model into another draft model (as an entirely separated model) with the word "Clone" at the end of its name
• Cancel Revision - Deletes the revision entirely and restores the old version of the model (moves back any suggestions associated with new version)

What can the user do

• Create any highlevel concept or field without any version changes. Names of created highlevel concepts or fields can also be changed
• Update description, standards mapping, related concepts, related terms and metadata. On update the version of the concept and its branch (ancestors and descendants) are changed without any version changes
• Change model description and standards without any version changes
• Deprecate/ Delete a highlevel concept or field without any version changes
• Import a highlevel concept from another data model with its fields without any version changes. If its fields are related to other highlevel concepts, those highlevel concepts will also be imported and so on.

What does this mean for ongoing mappings

• Finalized mappings are not affected
• New mapping created while a model is in review, still uses the stable version of the model and not the in review version
• Mappings that are half done and not finalized, will go through an upgrade process once the In Review model is published. What this means is that the user will be forced to upgrade the mapping to the latest version, which might cause the loss of some mappings (if the concept has been deprecated)

Deprecated

Is model available for use? No

Available model actions

• None

What can the user do

• Just view the model

Available Functionality

Create or clone concepts

Create a concept or clone a whole model.

Retrieve necessary data about concepts

Retrieve data models, data models with specific status, the major version of a model, concepts, the metadata of a concept, the uids of concepts or the root concept of a model.

Update concepts

Update a concept or publish it.

Delete or deprecate a concept

Delete a concept if it's in Draft or UnderRevision status or deprecate it if it's in Stable status.

Import (high-level concept) from another Data Model

On high-level concept creation, the functionality to copy the fields (along with all their related high-level concepts and their fields) of a high-level concept of a Stable model exists. In this case, the uids of those copied high-level concepts/ fields will remain the same, but their domain uids will be updated with the uid of the model (i.e. root concept) they are imported into.

Import Model

The Import Model functionality is only available to any user with the admin role assigned. In order for a model to be imported, its root concept should not have the same domainUid and uid with an already existing model (even if that model is deprecated).

Export Model

The Export Model functionality is only available to any user with the admin role assigned. Only Stable Models can be exported in a .json format.

Data Model Controller Caching

Any APIs used by the model manager UI we currenlty have no caching on, to make sure the model moderator always get the latest version of the model.

For all other "normal" usages we do two forms of caching:

Client Caching - We cache for 5 minutes on the client side. This is the most effiecient form of caching since there is no contact with the backend during these 5 minutes and the data loads in a matter of milliseconds. The time is limited to make sure that we take an updated version of the model, not too long after

Backend Caching - This caches the response at the level of the controller. Any resource using the CachingInterceptor has a specified expiration duration (currently set to a day). During the caching period we invalidate the cache at the moment on any model manager change event. This means that on any model creation/ update/ deprecation/ deletion we invalidate all caching of the data model resources. Using the new interceptor we also make sure that cache keys are unique based on the url parameters to make sure we don't end up in inconsistent states.

Available Services

checkNameValidity

Detects changes between existing concept and incoming update object.

Param	Type	Description
name	string	The name of a concept/ reference concept or prefix
type	string	The type of the field being validated (could be concept,reference-concept or prefix)

getDataModel

Retrieves all active data model trees.

getActiveRootConcepts

Retrieves all models indepedent of status.

getAllRootConcepts

Retrieves all models independent of status with their domain major version, if required

Param	Type	Description
includeDomainMajorVersion	boolean	Defaults to false. If true, the domain major version of each model is also returned

getModelConcepts

Retrieves all concepts of a given model.

Param	Type	Description
conceptId	number	the model id
status	Status	optionally a status to filter by

export enum Status {
    Draft = 'draft',
    UnderRevision = 'underRevision',
    Stable = 'stable',
    Deprecated = 'deprecated',
}

getConcept

Retrieves the specific concept by its id.

Param	Type	Description
conceptId	number	The id of the concept to retrieve

getDirectDescendantsCount

Retrieves a count of all direct children of a given concept

Param	Type	Description
parentConcept	ModelConcept	model concept to query

ModelConcept {
    id: number;
    uid: string;
    majorVersion: number;
    name: string;
    description: string;
    version: string;
    dateAdded = new Date();
    dateDeprecated: Date;
    type: string;
    standardsMapping: any;
    tokenizedTerms: any;
    relatedTerms: any;
    metadata: any;
    children: ModelConcept[];
    parent: ModelConcept;
    parentId: number;
    referenceConcept: ModelConcept;
    referenceConceptName: string;
    referenceConceptId: number;
    referencePrefix: any;
    status: string;
    updatedAt = new Date();
    domainUid: string;
}

getLeafsCount

Retrieves a count of all leaf concepts of given concept.

Same parameter as the getDirectDescendantsCount service.

getFlatModel

Retrieves a flat list of all concepts under model

Param	Type	Description
conceptId	number	the model id

getSubModel

Retrieves a sub tree of given concept with it's domain major version.

Param	Type	Description
conceptId	number	the id of the concept to get the sub tree for
status	Status	the status to optionally filter by

copyAndFindReferences

Performs copying of concepts and of their related concepts, if any.

Param	Type	Description
manager	EntityManager	transaction manager
concept	ModelConcept	concept to be cloned
initialParent	any	the concept whith is being copied
parentId	any	the parent id of the concept too be cloned

handleConceptCreation

Handles whether the create concept function should be called with the previous (from an on-going transcation) transcation manager or a new one.

Param	Type	Description
conceptData	CreateModelConceptDTO	Data to create new concept
previousManager?	EntityManager	optional transaction manager

export class CreateModelConceptDTO {
    name: string;
    description: string;
    type: string;
    standardsMapping: any;
    relatedTerms: any;
    metadata: any;
    parentId: number;
    referenceConceptId: number;
    referencePrefix: any;
    conceptToBeCopiedId: number;
    domainUid: string;
}

createConcept

Create a new concept (model/highlevel/field) based on the provided concept data.

For all concepts we validate based on their category (model, highlevel etc) that the given data is the expected ones.

• If concept is a model we create it as draft and give it the default starting version
• If concept belongs to a draft concept then itself is draft and takes the version of its parent
• If concept belongs to an in review concept then its a draft and takes the version of its parent
• If concept belongs to a stable concept then itself becomes stable, increases its minor version by one and propagates it to all its parents
• Deprecated parents can get concept children
• Validation check is performed for concept name, reference concept name, and prefixes in order to ensure that they include only characters and numbers and not special characters

Param	Type	Description
conceptData	CreateModelConceptDTO	Data to create new concept
manager	EntityManager	transaction manager

cloneAndCopyConcept

Copies and Creates a new concept (highlevel) and all of its fields and related concepts based on the provided concept data (which include the information which concept to copy from the other model). If required, the model in which the concept is going to be copied to is first cloned and then the concept is created and copied. If the model is first cloned then, the cloned model is returned, otherwise the created high level concept.

For all concepts we validate based on their category (i.e. highlevel in this case) that the given data is the expected ones.

• If model is Stable, a new cloned major version of the model is created, which takes the status under revision. We move any suggestions to the clone.
• If concept belongs to a draft model then itself is draft and takes the version of its parent
• If concept belongs to an in review model then its a draft and takes the version of its parent
• Deprecated parents can get concept children

Param	Type	Description
conceptData	CreateModelConceptDTO	Data to create new concept with the extra information, which is the concept to be copied from the other model

updateConcept

Update an existing concept (model/highlevel/field) based on the provided concept data.

For all concepts we validate based on their category (model, highlevel etc) that the given data is the expected ones.

• If changes are on a stable concept then we increament the patch version of the concept and its parents and siblings
• We update only the changes and inform the predictor accordingly both of the changes and new standards
• Deprecated concepts cannot be updated
• Models/concepts under review cannot be updated
• Validation check is performed for concept name, reference concept name, and prefixes in order to ensure that they include only characters and numbers and not special characters

Param	Type	Description
id	number	Id of concept to be updated
conceptData	UpdateModelConceptDTO	Data to update an existing concept

export class UpdateModelConceptDTO {
    name: string;
    description: string;
    type: string;
    standardsMapping: any;
    relatedTerms: any;
    metadata: any;
    referenceConceptId: number;
    referencePrefix: any;
}

publishConcept

Publish a model with the given id.

• Only draft and under revision models can be published
• Only models can be published
• If this is a draft model then we increment the model major version by 1 and propagate throughout the model. We send the creation of the model to the predictor along with the models it has across the entire model
• If this is an under revision model then we figure out the mapping between old and new ids, deprecated, new and existing fields, inform the predictor for each of these. Then we deprecate the old model and set the new one to stable

Param	Type	Description
id	number	Id of model to be published

deleteConcept

Deletes/deprecates the given id.

• If the concept is draft then we just delete it
• If the concept is under revision and is a model then we move any suggesions back to the previous version and then delete it entirely
• If it's under revision and not a model then we mark it as deprecated
• If it's stable and a field/related field then we mark it deprecated and increase the minor version
• If it's a model then we just mark it as deprecated
• If it's a high level concept we create a clone of the model. The clone takes the status under revision. We move any suggestions to the clone. We mark in the old model the next version id using the clone id

Param	Type	Description
id	number	Id of concept to be deleted/deprecated

cloneConcept

Clones a model into a brand new one.

• No association with previous model
• Name generated to be unique

Param	Type	Description
id	number	The id of the model to clone

getMappingToNewModel

Retrieves a mapping of old version ids to latest model concept object.

Param	Type	Description
id	number	Id of the old model to get the mapping for

associateReference

A helper function that associates a given concept to a concept provided a string path by finding the id of the corresponding concept to that path.

Param	Type	Description
manager	EntityManager	transaction manager
curConcept	ModelConcept	concept to add reference to
path	string	string path of concept to associate to
assConceptId	number	reference concept id
assConceptName	string	reference concept name

cloneModel

Clones a give model making sure to copy all concepts while maintaining tree structure and references.

As part of the cloning process you can provide a changes object that contains a map of things you want changed in the new instance. For example providing changes {version: '1.0.0', status: (c: ModelConcept)=> return 'Deprecated'} will cause all models to get a version 1.0.0 and the status to become deprecated. Using the function approach you can add conditions based on the model concept and if you want the status to be changed and to what.

Param	Type	Description
manager	EntityManager	transaction manager
concept	ModelConcept	model to clone
path	string	path to model to be cloned
changes	any	any changes we want applied to the cloned model
caller	string	indication from which root function this was called
deletions	any = []	the ids of any concepts you want deleted
moveSuggestions	boolean = false	indication from which root function this was called
keepDeprecated	boolean = false	any changes we want applied to the cloned model

moveSuggestions

Moves the suggestions of a given model to another.

Param	Type	Description
manager	EntityManager	transaction manager
idMap	any	a map of old ids mapping to the new concepts
fromId	number	id of the model we are copying suggestions from
toId	number	id of the model we are copying suggestions to

shouldDelete

Helper function that checks against a list of deletions and returns whether it should be deleted or not.

Param	Type	Description
concept	ModelConcept	the concept to check
deletions	deletions: { id?: number; referenceConceptId?: number }[]	list of deletion rules

cloneAndFindReferences

Performs main cloning and returns

• the cloned model concept
• the idMap of old ids to new concepts
• any references found that need to be associated

Param	Type	Description
manager	EntityManager	transaction manager
concept	ModelConcept	concept to be cloned
path	string	the path of the concept to be cloned
parentId	number	null
changes	any	any changes to be performed on the cloned object
caller	string	indication from which root function this was called
deletions	{ id?: number; referenceConceptId?: number }[]	the ids of any concepts you want deleted
keepDeprecated	boolean	any changes we want applied to the cloned model
idMap	{}	any mappings already made between old ids an new concepts
references	{ from: string; to: number }[] = []	any references found up to now

deleteClosure

Handles deleting a closure of a given concept. It removes any closures that have the concept as an ancestor or as a descendant.

Param	Type	Description
manager	EntityManager	transaction manager
concept	ModelConcept	the concept to delete its closure

deleteReferences

Handles deleting any references of a given concept.

Param	Type	Description
manager	EntityManager	transaction manager
concept	ModelConcept	the concept to delete its references

deleteAllChildren

Deletes all children of a given concept. Makes sure to delete:

• child references
• child children
• where the child is referenced as a next version
• any child closures
• the child itself

Param	Type	Description
manager	EntityManager	transaction manager
concept	ModelConcept	the concept to delete its children

bubbleUp

Propagates any changes to attributes provided in the changes object to all the ancestors of the provided concept.

Param	Type	Description
manager	EntityManager	transaction manager
concept	ModelConcept	The child concept that has the change
changes	any	What changes are to be applied
applyToSelf	boolean = false	If they should be applied to the concept itself

updateAndPropagate

Propagates any changes to attributes provided in the changes object to all the descednatns of the provided concept.

Param	Type	Description
manager	EntityManager	transaction manager
concept	ModelConcept	The child concept that has the change
changes	any	What changes are to be applied

updateConceptAttributes

Given a list of changes it applies them to the given concept.

Param	Type	Description
manager	EntityManager	transaction manager
concept	ModelConcept	The concept to update
changes	any	The changes to perform

findNumberOfLeafs

Counts the number of leaves of a concept based on the statuses provided.

Param	Type	Description
tree	ModelConcept	Model concept
statusesToFilterBy	string[]	Statuses to include

statusesToCheckBasedOnConceptStatus

Returns a list of status to be used to filter concepts for a given concept. For example if a concept is draft then we should consider both stable and draft concepts when returning from the API.

Param	Type	Description
concept	ModelConcept	Concept to consider

getRoot

Retrieves the top parent of a given concept.

Param	Type	Description
concept	ModelConcept	The concept to find the root of

siblingWithName

Finds any sibling with the same name.

Param	Type	Description
manager	EntityManager	transaction manager
conceptName	string	Name to check
concept?	ModelConcept	Concept to check if its an existing one
parent?	ModelConcept	Concept parent

hasChanged

Detects if given object changed given before and after states.

Param	Type	Description
before	any	Object prior state
after	any	Object new state

findChanges

Detects changes between existing concept and incoming update object.

Param	Type	Description
fields	{ field: ConceptField; required: boolean }[]	Fields to check
concept	ModelConcept	Existing concept
conceptData	UpdateModelConceptDTO	Updated data

export class UpdateModelConceptDTO {
    name: string;
    description: string;
    type: string;
    standardsMapping: any;
    relatedTerms: any;
    metadata: any;
    referenceConceptId: number;
    referencePrefix: any;
}

validateConceptFields

Validates the changes to be applied to existing concept.

Param	Type	Description
strict	boolean	strict mode is true in case the concept in a status where we want strict non-draft checks to apply (for example checking that certain metadata don't switch from one state to another)
fields	{ field: ConceptField; required: boolean }[]	the fields to be validated depending on the status
conceptData	UpdateModelConceptDTO	The updated data
concept	ModelConcept	the concept to update

getParentConceptFromId

Gets the parent object from the DTO.

Param	Type	Description
manager	EntityManager	transaction manager
conceptData	CreateModelConceptDTO	The DTO data to check in

export class CreateModelConceptDTO {
    name: string;
    description: string;
    type: string;
    standardsMapping: any;
    relatedTerms: any;
    metadata: any;
    parentId: number;
    referenceConceptId: number;
    referencePrefix: any;
    conceptToBeCopiedId: number;
    domainUid: string;
}

getReferenceConceptFromId

Gets the referenced object from the DTO.

Param	Type	Description
manager	EntityManager	transaction manager
conceptData	CreateModelConceptDTO (Same as the service above)	The DTO data to check in

getFullFlatModel

Retrieves the entire model in a flat list.

Param	Type	Description
manager	EntityManager	transaction manager
model	ModelConcept	The model to retrieve
includeDeprecated	boolean = true	Check to include or not deprecated concepts

getAllModelStandards

Retrieves all standards used in the model itself and children.

Param	Type	Description
manager	EntityManager	transaction manager
model	ModelConcept	Model to get standards from

sendPredictorNewStandards

Sends the predictor only new standards created in updated concept that do not exist anywhere else in the model tree.

Param	Type	Description
manager	EntityManager	transaction manager
model	ModelConcept	The model for which to send the standards for
concept	ModelConcept	The changed concept

getConceptInTransaction

Finds a concept in the transaction by id.

Param	Type	Description
manager	EntityManager	transaction manager
conceptId	number	Concept Id

previousVersion

Retrieves the previous version of a concept if it exists.

Param	Type	Description
manager	EntityManager	transaction manager
concept	number	Concept to find previous version of

getAfterBeforeMap

Retrieves a map of new version id as the key and the previous version as the value.

Param	Type	Description
manager	EntityManager	transaction manager
beforeConcept	ModelConcept	The previous concept or model to create the map of

filterTrees

Filters a list of trees by status.

Param	Type	Description
trees	ModelConcept[]	The trees to filter
status	Status	The status to filter by

export enum Status {
    Draft = 'draft',
    UnderRevision = 'underRevision',
    Stable = 'stable',
    Deprecated = 'deprecated',
}

filterTree

Filters a tree by status.

Param	Type	Description
tree	ModelConcept	The tree to filter
status	Status (Same as service above)	The status to filter by

filterChildren

Filters children by status.

Param	Type	Description
children	ModelConcept[]	List of children
status	Status (Same as service above)	Status to filter by

getLatestVersion

Retrieves the latest version of the given concept uid and domain uid.

Param	Type	Description
conceptUid	string	The concept uid
domainUid	string	The concept domain uid

nextVersion

Retrieves the next version of a concept if it exists.

Param	Type	Description
manager	EntityManager	transaction manager
concept	ModelConcept	Concept to find next version of

getConceptsFromIDs

Retrieves concepts by their ids.

Param	Type	Description
ids	Array<number>	Array of concept ids

exportModel

Export model by id.

Param	Type	Description
id	number	Id of model to be exported

createImportedConcepts

Create imported concepts.

Param	Type	Description
manager	EntityManager	transaction manager
concept	any	Concept to be created
parentId	number	null
changes	any	any changes to be performed on the concept to be created
references	{ from: number; to: number }[] = []	the mapping between the old ids and the new created ids of the high level concepts

importModel

Import model.

Param	Type	Description
data	any	Model data to be imported