Databricks AI Models Collibra Integration – Easy Connections

Easy Connections is an Expleo’s integration platform which runs on Customer’s environment to integrate various source systems to Collibra.

Databricks Connector runs on top of this Easy Connections platform to integrate metadata, metrics and lineage to datasets used for training AI Models in Databricks, into Collibra Platform. 

Use Case(s) 

• Model Catalog – A comprehensive inventory of models developed using Databricks, including associated metrics and metadata.  

• Model Lifecycle Management – Utilizing the metadata from the cataloged AI/ML models to orchestrate workflows that support the end-to-end management of model lifecycles.  

• Traceability and Lineage – Enables tracking of a model’s origin, including the datasets used for training. This lineage information helps organizations identify whether models involve sensitive or personally identifiable information (PII), allowing them to enforce appropriate security and compliance policies.

Below are some of the key features of the connector 

• Ingest historic run and experiment metadata to understand how the model has evolved. 

• Various authentication mechanism: 

• • PAT token 

• • Username and password 

• • Client ID, Client Secret, and Tenant ID based. 

• Captures enhanced metrics (custom or user defined metrics) logged in Databricks Notebooks 

• Lineage to datasets (RDBMS/Files) used for training the models. 

• Both Workspace registered and Unity Catalog registered models can be synced. 

• Filtering based on Workspace and AI Models in Databricks. 

• Modification and Delete of the source (Databricks) objects are automatically captured and can be used to trigger actions in Collibra such as asset status change, Model Lifecycle management workflows, etc. 

• Support for full load and incremental load. Incremental loads can be based on the last successful execution or from a specific date 

Refer to the attached datasheet for detailed information on supported and unsupported features