Unsupported Screen Size: The viewport size is too small for the theme to render properly.

Populate Data Warehouse with Daily Snapshots of Collibra Analytical Data

Published by: Collibra Marketplace
Latest version: 1.0.3
Released: November 25, 2022
Contact Publisher
PackageDocumentation
Community Offering

Your use of Community Offerings is subject to the Collibra Marketplace License Agreement. Read more.

Overview

“Community Offerings” are offerings published by third parties or Collibra within the Collibra Marketplace community.  Community Offerings are free and made available “as is” by Collibra.  Your master commercial agreement with Collibra for the Collibra Service DOES NOT apply to your use of the Community Offerings (including any warranties, support services and service levels referenced therein).   Your use of Community Offerings is subject to the Collibra Marketplace License Agreement, as may be modified from time to time by Collibra.  Collibra encourages users to seek assistance for Community Offerings from peers and other Data Citizens via the Collibra Community platform.

Many integrations around the Collibra platform require data from Collibra to be fed to other systems for a number of different reasons. Whether for reporting purposes around the content and business processes managed in Collibra or for provisioning curated or governed content to downstream systems, an integration may require access to daily snapshots of the Collibra content to fulfil a particular requirement. This integration template demonstrates how to:

  • Obtain a daily data snapshot of the data from Collibra using the Reporting Data Layer as part of the Collibra Insights product.
  • Decompress and parse the daily data snapshot.
  • Import into a data warehouse such as Google BigQuery.

This Spring Boot-based integration template uses the Collibra REST Core API to obtain the daily data snapshot generated by the Reporting Data Layer. The information is returned as a ZIP file that is decompressed in a temporary folder, which contains a number of Parquet files. The whole folder structure is then imported into BigQuery. This integration template also exposes an endpoint that enables you to perform SQL queries over the imported data in Google BigQuery.

Elements in Scope

This integration does not access a Collibra environment directly. It uses the Collibra REST Core API to retrieve the Reporting Data archive file that contains Apache Parquet files with table content for each of the six concepts (for one day, which is the snapshot date):

  • community
  • domain
  • asset
  • attribute
  • relation
  • responsibility

To receive support on this item, you can engage our Professional Services team or post any questions in the Data Citizens Community.

Media

More details

Release Notes
  • Removed security vulnerabilities
  • Upgraded to Spring Boot version 2.7.5
  • Docker file added
Compatibility
  • Spring Boot Framework
  • Eclipse IDE
  • Collibra Data Intelligence Cloud
  • Collibra Data Intelligence On-Prem
Dependency
  • Java Runtime Environment 1.8
  • Spring Boot Integration Library
  • Collibra Data Intelligence Cloud API v2
  • Collibra Reporting Data Layer
License and Usage Requirements

Release History

Version 1.0.2
April 7, 2022
Release Notes
  • Updated the Spring Boot Starter Parent version to 2.5.12
  • Updated the Collibra Integration Library version to 1.1.5
Compatibility
  • Spring Boot Framework
  • Eclipse IDE
  • Collibra Data Intelligence Cloud
  • Collibra Data Intelligence On-Prem
Dependency
  • Java Runtime Environment 1.8
  • Spring Boot Integration Library
  • Collibra Data Intelligence Cloud API v2
  • Collibra Reporting Data Layer
License and Usage Requirements
Version 1.0.1
March 21, 2022
Release Notes

– Refactoring according to standard-reference template.
– File Structuring of Components and Services
– Constructor Autowiring
– Adding Constants
– Fixing Type Warnings
– Exception Handling
– Lombok
– Sync Lock
– Adding AppConfig
– Updating Authentication Config
– Main Service with ETL format

Compatibility
  • Spring Boot Framework
  • Eclipse IDE
  • Collibra Data Intelligence Cloud
  • Collibra Data Intelligence On-Prem
Dependency
  • Java Runtime Environment 1.8
  • Spring Boot Integration Library
  • Collibra Data Intelligence Cloud API v2
License and Usage Requirements
Version 1.0.0
September 3, 2021
Release Notes

Initial release:

Spring Boot integration that exports Reporting-Insights-Data from Collibra and imports this information to Google Cloud, it also exposes an endpoint to perform SQL queries on the imported data.

Compatibility
  • Spring Boot Framework
  • Eclipse IDE
  • Collibra Data Intelligence Cloud
Dependency
  • Java Runtime Environment 8
  • Collibra Data Intelligence Cloud API v2
  • Collibra Platform v2021+
  • Spring Boot Integration Library
License and Usage Requirements

See existing Q&A in the Collibra Community

Browse discussions with customers who also use Community Offerings from the Collibra Marketplace.

Start a New Topic in the Collibra Community

Collibra-hosted discussions connect you to other customers who use this app.

The following terms shall apply to the extent you receive the source code to this offering:  Notwithstanding the terms of the Binary Code License Agreement under which this integration template is licensed, Collibra grants you, the Licensee, the right to access the source code to the integrated template in order to copy and modify said source code for Licensee’s internal use purposes and solely for the purpose of developing connections and/or integrations with Collibra products and services.

Solely with respect to this integration template, the term “Software,” as defined under the Binary Code License Agreement, shall include the source code version thereof. Except with respect to the foregoing, all remaining terms of the Binary Code License Agreement shall apply to the license of integration template hereunder.

Furthermore, this listing requires a Collibra Insights listing to get access to the Reporting Data Layer.

Reviews

Rating
Leave a review