Unity Catalog to Collibra integration – metadata and lineage tracking
Overview
Creates, updates and represents existing Unity Catalog metastore, catalog, schema, table and column resources in Collibra.
A metadata archive a Spring Boot integration that takes requests from Collibra and consume Unity Catalog and Unity Catalog Lineage Tracking REST API services to discover and register unity catalog metastores, catalogs, schemas, tables, columns and its dependencies. At the time of this submission, Unity Catalog was in Private Preview and the Unity Catalog and Unity Catalog Lineage Tracking REST APIs were limited in what it could offer. The following areas are not covered by this document today but are in scope of future releases:
Delta Sharing APIs
Databricks-internal APIs
e.g. all related Column ACL
Requirements and user stories:
- As a data engineer I want to give my data steward and data users full visibility of your databricks metastore resources by bringing metadata into a central location, making data available and easily accessible across your organization.
- As a data steward i want to improve data transparency by helping establish a one enterprise-wide repository of assets, so every user can easily understand and discover data relevant to them.
Databricks regularly provides previews to give you a chance to evaluate and provide feedback on features before they’re generally available (GA). These preview releases can come in various degrees of maturity, each of which is defined in this article. For more information about Databricks Runtime releases, including support lifecycle and long-term-support (LTS), see Databricks runtime support lifecycle.
Mar 2022 update: Unity Catalog is now in gated public preview. During this gated public preview, Unity Catalog has the following limitations.
-
Python, Scala, and R workloads are supported only on Data Science & Engineering or Databricks Machine Learning clusters that use the Single User security mode and do not support dynamic views for the purpose of row-level or column-level security.
-
Unity Catalog can be used together with the built-in Hive metastore provided by Databricks. External Hive metastores that require configuration using init scripts are not supported.
-
Overwrite mode for dataframe write operations into Unity Catalog is supported only for managed Delta tables and not for other cases, such as external tables. In addition, the user must have the CREATE privilege in the parent schema and must be the owner of the existing object.
-
Service Principals are not supported as account-level identities.
Please refer to Unity Catalog Preview Limitations for more information.
May 2022 update: Welcome to the Data Lineage Private Preview! Unity Catalog now captures runtime data lineage for any table to table operation executed on a Databricks cluster or SQL endpoint. Lineage is captured at the granularity of tables and columns, and the service operates across all languages.
June 2022 update: Unity Catalog Lineage is now captured and catalogued both as asset relations and as custom technical lineage.
July 2022 update: Unity Catalog API will be switching from v2.0 to v2.1 as of Aug 11, 2022, after which v2.0 will no longer be supported.
Media
More details
Release Notes
Version 1.0.7 will allow to extract metadata from databricks with non-admin Personal Access Token.
Compatibility
- Springboot Framework
- Lineage Tracking API
- Unity Catalog API
- Collibra Data Intelligence Cloud
Dependency
- Lineage Tracking API
- Unity Catalog API
- Springboot Framework 2.6.6
- Java Development Kit
License and Usage Requirements
- Collibra Catalog
Release History
Release Notes
the new release version 1.0.6 is for enhancing the application to accept wildcard character as part of schema names.
Compatibility
- Springboot Framework
- Unity Catalog API
- Lineage Tracking API
- Collibra Data Intelligence Cloud
Dependency
- Unity Catalog API
- Lineage Tracking API
- Java Development Kit
- Springboot Framework 2.6.6
License and Usage Requirements
- Collibra Catalog
Release Notes
Release to update the Spring Boot App for the changes in Databricks Unity Catalog API
Compatibility
- Springboot Framework
- Unity Catalog API
- Lineage Tracking API
- Collibra Data Intelligence Cloud
Dependency
- Unity Catalog API
- Lineage Tracking API
- Java Development Kit
- Springboot Framework 2.6.6
License and Usage Requirements
- Collibra Catalog
Release Notes
Unity Catalog API will be switching from v2.0 to v2.1 as of Aug 11, 2022, after which v2.0 will no longer be supported.
Compatibility
- Springboot Framework
- Unity Catalog API
- Lineage Tracking API
- Collibra Data Intelligence Cloud
Dependency
- Unity Catalog API
- Lineage Tracking API
- Java Development Kit 1.11
- Springboot Framework 2.6.6
License and Usage Requirements
- Collibra Catalog
Release Notes
June 2022 updated: Unity Catalog Lineage is now captured and catalogued both as asset relations and as custom technical lineage.
Compatibility
- Springboot Framework
- Unity Catalog API
- Lineage Tracking API
- Collibra Data Intelligence Cloud
Dependency
- Unity Catalog API
- Java Development Kit 1.11
- Springboot Framework 2.6.6
License and Usage Requirements
- Collibra Catalog
Release Notes
May 2022 update: Welcome to the Data Lineage Private Preview! Unity Catalog now captures runtime data lineage for any table to table operation executed on a Databricks cluster or SQL endpoint. Lineage is captured at the granularity of tables and columns, and the service operates across all languages.
Compatibility
- Spring Boot Framework
- Unity Catalog API
- Lineage Tracking API
- Collibra Data Intelligence Cloud
Dependency
- Unity Catalog API
- Java Development Kit v1.11
- Spring Boot framework v2.6.6
License and Usage Requirements
- Collibra Catalog
To receive support on this item, you can engage our Professional Services team or post any questions in the Data Citizens Community.
The following terms shall apply to the extent you receive the source code to this offering.Notwithstanding the terms of the Binary Code License Agreement under which this integration template is licensed, Collibra grants you, the Licensee, the right to access the source code to the integrated template in order to copy and modify said source code for Licensee’s internal use purposes and solely for the purpose of developing connections and/or integrations with Collibra products and services.Solely with respect to this integration template, the term “Software,” as defined under the Binary Code License Agreement, shall include the source code version thereof. Except with respect to the foregoing, all remaining terms of the Binary Code License Agreement shall apply to the license of integration template hereunder.
