Unsupported Screen Size: The viewport size is too small for the theme to render properly.

Collibra and Unity for catalog and lineage tracking

Published by: Collibra Marketplace
Latest version: 1.0.2
Released: April 8, 2022
Package Documentation
Community Offering

Your use of Community Offerings is subject to the Collibra Marketplace License Agreement. Read more.

Overview

Creates, updates and represents existing Unity Catalog metastore, catalog, schema, table and column resources in Collibra.

A metadata archive a Spring Boot integration that takes requests from Collibra and consume Unity Catalog REST API services to discover and register unity catalog metastores, catalogs, schemas, tables and columns. At the time of this submission, Unity Catalog was in Private Preview and the UC REST API was limited in what it could offer. The following areas are not covered by this document today but are in scope of future releases:

Delta Sharing APIs

Databricks-internal APIs

e.g. all related  Column ACL

Requirements and user stories:

  • As a data engineer I want to give my data steward and data users full visibility of your databricks metastore resources by bringing metadata into a central location, making data available and easily accessible across your organization.
  • As a data steward i want to improve data transparency by helping establish a one enterprise-wide repository of assets, so every user can easily understand and discover data relevant to them.

At the time of this submission, Unity Catalog was in Public Preview and the UC REST API was limited in what it could offer. Databricks regularly provides previews to give you a chance to evaluate and provide feedback on features before they’re generally available (GA). These preview releases can come in various degrees of maturity, each of which is defined in this article. For more information about Databricks Runtime releases, including support lifecycle and long-term-support (LTS), see Databricks runtime support lifecycle.

Mar 2022 update: Unity Catalog is now in gated public preview. During this gated public preview, Unity Catalog has the following limitations.

  • Python, Scala, and R workloads are supported only on Data Science & Engineering or Databricks Machine Learning clusters that use the Single User security mode and do not support dynamic views for the purpose of row-level or column-level security.

  • Unity Catalog can be used together with the built-in Hive metastore provided by Databricks. External Hive metastores that require configuration using init scripts are not supported.

  • Overwrite mode for dataframe write operations into Unity Catalog is supported only for managed Delta tables and not for other cases, such as external tables. In addition, the user must have the CREATE privilege in the parent schema and must be the owner of the existing object.

  • Service Principals are not supported as account-level identities.

Please refer to Unity Catalog Preview Limitations for more information.

Apr 2022 update: Welcome to the Data Lineage Private Preview! Unity Catalog now captures runtime data lineage for any table to table operation executed on a Databricks cluster or SQL endpoint. Lineage is captured at the granularity of tables and columns, and the service operates across all languages.

 

Media

More details

Release Notes

Apr 2022 update: Welcome to the Data Lineage Private Preview! Unity Catalog now captures runtime data lineage for any table to table operation executed on a Databricks cluster or SQL endpoint. Lineage is captured at the granularity of tables and columns, and the service operates across all languages.

Compatibility
  • Spring Boot Framework
  • Unity Catalog API
  • Lineage Tracking API
  • Collibra Cloud
Dependency
  • Unity Catalog API
  • Java Development Kit v1.11
  • Spring Boot framework v2.6.6
License and Usage Requirements
  • Collibra Catalog

Release History

No previous versions of this listing is available.

To receive support on this item, you can engage our Professional Services team or post any questions in the Data Citizens Community.
The following terms shall apply to the extent you receive the source code to this offering.Notwithstanding the terms of the Binary Code License Agreement under which this integration template is licensed, Collibra grants you, the Licensee, the right to access the source code to the integrated template in order to copy and modify said source code for Licensee’s internal use purposes and solely for the purpose of developing connections and/or integrations with Collibra products and services.Solely with respect to this integration template, the term “Software,” as defined under the Binary Code License Agreement, shall include the source code version thereof. Except with respect to the foregoing, all remaining terms of the Binary Code License Agreement shall apply to the license of integration template hereunder.

Reviews

Rating
Leave a review