Populate Data Warehouse with Daily Snapshots of Collibra Analytical Data
Overview
Many integrations around the Collibra platform require data from Collibra to be fed to other systems for a number of different reasons. Whether for reporting purposes around the content and business processes managed in Collibra or for provisioning curated or governed content to downstream systems, an integration may require access to daily snapshots of the Collibra content to fulfil a particular requirement. This integration template demonstrates how to:
- Obtain a daily data snapshot of the data from Collibra using the Reporting Data Layer as part of the Collibra Insights product.
- Decompress and parse the daily data snapshot.
- Import into a data warehouse such as Google BigQuery.
This Spring Boot-based integration template uses the Collibra REST Core API to obtain the daily data snapshot generated by the Reporting Data Layer. The information is returned as a ZIP file that is decompressed in a temporary folder, which contains a number of Parquet files. The whole folder structure is then imported into BigQuery. This integration template also exposes an endpoint that enables you to perform SQL queries over the imported data in Google BigQuery.
Elements in Scope
This integration does not access a Collibra environment directly. It uses the Collibra REST Core API to retrieve the Reporting Data archive file that contains Apache Parquet files with table content for each of the six concepts (for one day, which is the snapshot date):
- community
- domain
- asset
- attribute
- relation
- responsibility
To receive support on this item, you can engage our Professional Services team or post any questions in the Data Citizens Community.
Media
More details
Release Notes
- Removed security vulnerabilities
- Upgraded to Spring Boot version 2.7.5
- Docker file added
Compatibility
- Spring Boot Framework
- Eclipse IDE
- Collibra Data Intelligence Cloud
- Collibra Data Intelligence On-Prem
Dependency
- Java Runtime Environment 1.8
- Spring Boot Integration Library
- Collibra Data Intelligence Cloud API v2
- Collibra Reporting Data Layer
License and Usage Requirements
Release History
Release Notes
- Updated the Spring Boot Starter Parent version to 2.5.12
- Updated the Collibra Integration Library version to 1.1.5
Compatibility
- Spring Boot Framework
- Eclipse IDE
- Collibra Data Intelligence Cloud
- Collibra Data Intelligence On-Prem
Dependency
- Java Runtime Environment 1.8
- Spring Boot Integration Library
- Collibra Data Intelligence Cloud API v2
- Collibra Reporting Data Layer
License and Usage Requirements
Release Notes
– Refactoring according to standard-reference template.
– File Structuring of Components and Services
– Constructor Autowiring
– Adding Constants
– Fixing Type Warnings
– Exception Handling
– Lombok
– Sync Lock
– Adding AppConfig
– Updating Authentication Config
– Main Service with ETL format
Compatibility
- Spring Boot Framework
- Eclipse IDE
- Collibra Data Intelligence Cloud
- Collibra Data Intelligence On-Prem
Dependency
- Java Runtime Environment 1.8
- Spring Boot Integration Library
- Collibra Data Intelligence Cloud API v2
License and Usage Requirements
Release Notes
Initial release:
Spring Boot integration that exports Reporting-Insights-Data from Collibra and imports this information to Google Cloud, it also exposes an endpoint to perform SQL queries on the imported data.
Compatibility
- Spring Boot Framework
- Eclipse IDE
- Collibra Data Intelligence Cloud
Dependency
- Java Runtime Environment 8
- Collibra Data Intelligence Cloud API v2
- Collibra Platform v2021+
- Spring Boot Integration Library
License and Usage Requirements
See existing Q&A in the Data Citizens Community
Browse discussions with customers who also use Community Offerings from the Collibra Marketplace.
Start a New Topic in the Data Citizens Community
Collibra-hosted discussions connect you to other customers who use this app.
The following terms shall apply to the extent you receive the source code to this offering: Notwithstanding the terms of the Binary Code License Agreement under which this integration template is licensed, Collibra grants you, the Licensee, the right to access the source code to the integrated template in order to copy and modify said source code for Licensee’s internal use purposes and solely for the purpose of developing connections and/or integrations with Collibra products and services.
Solely with respect to this integration template, the term “Software,” as defined under the Binary Code License Agreement, shall include the source code version thereof. Except with respect to the foregoing, all remaining terms of the Binary Code License Agreement shall apply to the license of integration template hereunder.
Furthermore, this listing requires a Collibra Insights listing to get access to the Reporting Data Layer.
