Unsupported Screen Size: The viewport size is too small for the theme to render properly.

Unity Catalog to Collibra integration – metadata and lineage tracking

Published by: Collibra Marketplace
Latest version: 1.0.7
Released: September 1, 2022
Package Documentation
Community Offering

Your use of Community Offerings is subject to the Collibra Marketplace License Agreement. Read more.

Overview

For this specific integration (and all other Custom Integrations listed on the Collibra Marketplace), please read the following disclaimer:

  • This integration is a template that has been developed in cooperation with a few select clients based on their custom use cases and business needs.
  • While all effort has been made to encompass a range of typical usage scenarios, specific needs beyond this may require chargeable template customization.
  • With this in mind, we have made sure that the template is available as source code and readily modifiable to suit the client's particular use case.

Creates, updates and represents existing Unity Catalog metastore, catalog, schema, table and column resources in Collibra.

A metadata archive a Spring Boot integration that takes requests from Collibra and consume Unity Catalog and Unity Catalog Lineage Tracking REST API services to discover and register unity catalog metastores, catalogs, schemas, tables, columns and its dependencies. At the time of this submission, Unity Catalog was in Private Preview and the Unity Catalog and Unity Catalog Lineage Tracking REST APIs were limited in what it could offer. The following areas are not covered by this document today but are in scope of future releases:

Delta Sharing APIs

Databricks-internal APIs

e.g. all related  Column ACL

Requirements and user stories:

  • As a data engineer I want to give my data steward and data users full visibility of your databricks metastore resources by bringing metadata into a central location, making data available and easily accessible across your organization.
  • As a data steward i want to improve data transparency by helping establish a one enterprise-wide repository of assets, so every user can easily understand and discover data relevant to them.

Databricks regularly provides previews to give you a chance to evaluate and provide feedback on features before they’re generally available (GA). These preview releases can come in various degrees of maturity, each of which is defined in this article. For more information about Databricks Runtime releases, including support lifecycle and long-term-support (LTS), see Databricks runtime support lifecycle.

Mar 2022 update: Unity Catalog is now in gated public preview. During this gated public preview, Unity Catalog has the following limitations.

  • Python, Scala, and R workloads are supported only on Data Science & Engineering or Databricks Machine Learning clusters that use the Single User security mode and do not support dynamic views for the purpose of row-level or column-level security.

  • Unity Catalog can be used together with the built-in Hive metastore provided by Databricks. External Hive metastores that require configuration using init scripts are not supported.

  • Overwrite mode for dataframe write operations into Unity Catalog is supported only for managed Delta tables and not for other cases, such as external tables. In addition, the user must have the CREATE privilege in the parent schema and must be the owner of the existing object.

  • Service Principals are not supported as account-level identities.

Please refer to Unity Catalog Preview Limitations for more information.

May 2022 update: Welcome to the Data Lineage Private Preview! Unity Catalog now captures runtime data lineage for any table to table operation executed on a Databricks cluster or SQL endpoint. Lineage is captured at the granularity of tables and columns, and the service operates across all languages.

June 2022 update: Unity Catalog Lineage is now captured and catalogued both as asset relations and as custom technical lineage.

July 2022 update: Unity Catalog API will be switching from v2.0 to v2.1 as of Aug 11, 2022, after which v2.0 will no longer be supported.

Media

More details

Release Notes

Version 1.0.7 will allow to extract metadata from databricks with non-admin Personal Access Token.

Compatibility
  • Springboot Framework
  • Lineage Tracking API
  • Unity Catalog API
  • Collibra Data Intelligence Cloud
Dependency
  • Lineage Tracking API
  • Unity Catalog API
  • Springboot Framework 2.6.6
  • Java Development Kit
License and Usage Requirements
  • Collibra Catalog

Release History

Version 1.0.6
August 30, 2022
Release Notes

the new release version 1.0.6 is for enhancing the application to accept wildcard character as part of schema names.

Compatibility
  • Springboot Framework
  • Unity Catalog API
  • Lineage Tracking API
  • Collibra Data Intelligence Cloud
Dependency
  • Unity Catalog API
  • Lineage Tracking API
  • Java Development Kit
  • Springboot Framework 2.6.6
License and Usage Requirements
  • Collibra Catalog
Version 1.0.5
August 16, 2022
Release Notes

Release to update the Spring Boot App for the changes in Databricks Unity Catalog API

Compatibility
  • Springboot Framework
  • Unity Catalog API
  • Lineage Tracking API
  • Collibra Data Intelligence Cloud
Dependency
  • Unity Catalog API
  • Lineage Tracking API
  • Java Development Kit
  • Springboot Framework 2.6.6
License and Usage Requirements
  • Collibra Catalog
Version 1.0.4
July 30, 2022
Release Notes

Unity Catalog API will be switching from v2.0 to v2.1 as of Aug 11, 2022, after which v2.0 will no longer be supported.

Compatibility
  • Springboot Framework
  • Unity Catalog API
  • Lineage Tracking API
  • Collibra Data Intelligence Cloud
Dependency
  • Unity Catalog API
  • Lineage Tracking API
  • Java Development Kit 1.11
  • Springboot Framework 2.6.6
License and Usage Requirements
  • Collibra Catalog
Version 1.0.3
June 28, 2022
Release Notes

June 2022 updated: Unity Catalog Lineage is now captured and catalogued both as asset relations and as custom technical lineage.

Compatibility
  • Springboot Framework
  • Unity Catalog API
  • Lineage Tracking API
  • Collibra Data Intelligence Cloud
Dependency
  • Unity Catalog API
  • Java Development Kit 1.11
  • Springboot Framework 2.6.6
License and Usage Requirements
  • Collibra Catalog
Version 1.0.2
June 10, 2022
Release Notes

May 2022 update: Welcome to the Data Lineage Private Preview! Unity Catalog now captures runtime data lineage for any table to table operation executed on a Databricks cluster or SQL endpoint. Lineage is captured at the granularity of tables and columns, and the service operates across all languages.

Compatibility
  • Spring Boot Framework
  • Unity Catalog API
  • Lineage Tracking API
  • Collibra Data Intelligence Cloud
Dependency
  • Unity Catalog API
  • Java Development Kit v1.11
  • Spring Boot framework v2.6.6
License and Usage Requirements
  • Collibra Catalog

To receive support on this item, you can engage our Professional Services team or post any questions in the Data Citizens Community.

The following terms shall apply to the extent you receive the source code to this offering.Notwithstanding the terms of the Binary Code License Agreement under which this integration template is licensed, Collibra grants you, the Licensee, the right to access the source code to the integrated template in order to copy and modify said source code for Licensee’s internal use purposes and solely for the purpose of developing connections and/or integrations with Collibra products and services.Solely with respect to this integration template, the term “Software,” as defined under the Binary Code License Agreement, shall include the source code version thereof. Except with respect to the foregoing, all remaining terms of the Binary Code License Agreement shall apply to the license of integration template hereunder.

Reviews

Rating
Leave a review