Unsupported Screen Size: The viewport size is too small for the theme to render properly.

DBT to Collibra Integration

Published by: Collibra Marketplace
Latest version: 1.1.6
Released: December 2, 2022
Package Documentation
Community Offering

Your use of Community Offerings is subject to the Collibra Marketplace License Agreement. Read more.

Overview

For this specific integration (and all other Custom Integrations listed on the Collibra Marketplace), please read the following disclaimer:

  • This integration is a template that has been developed in cooperation with a few select clients based on their custom use cases and business needs.
  • While all effort has been made to encompass a range of typical usage scenarios, specific needs beyond this may require chargeable template customization.
  • With this in mind, we have made sure that the template is available as source code and readily modifiable to suit the client's particular use case.

This Spring Boot integration extracts dbt metadata using GraphQL calls and dbt API calls. The integration then transforms the metadata and loads it to Collibra using Collibra API calls. This summarizes the simple flow of the ETL operations.

There are two main functionalities for this integration:

  • Compile and extract SQLs from the dbt Cloud and create SQL files written to the physical local disk, ready for ingestion by the technical lineage harvester.
  • Extract dbt project functionalities like models, exposures, metrics, tests, and metadata from the dbt Cloud, which are then transformed and loaded to Collibra.

In the case of models, the integration searches for an existing database domain named according to the format [ System asset name ] > [ Database ] > [ Schema ]. If the database exists, the integration updates it. Otherwise, it creates a new domain.

For dbt meta, the integration searches for attribute types by meta names given in the dbt project YML file. If there are none, the integration creates attribute types and loads assets with meta attributes. If you want to make this visible for the asset, you have to add characteristics from global assignments from the Collibra settings.

In the case of exposures, metrics, and tests, the integration looks for the name provided in the application properties file. If the domains exist, the integration updates them. Otherwise, new domains are created.

From the metrics, exposures, and tests GraphQL calls, the integration extracts metadata such as:

  • Dimensions for relations (it is used for the columns),
  • Filters (to apply to the underlying model for the metric),
  • SQL expression (to calculate the metrics).
  • Tests predicates
  • Database assets from models

A CMA file is also included. Once this CMA file is imported within your Collibra instance, it will ensure that all the custom prerequisites will be generated automatically.

Alternative ways to try out this integration are via the supplied JAR file or Docker image with the compiled build version of the integration. These contain all the required dependencies and source code. After a minimal number of required changes as described in this document, users can run the integration using one simple command. Please refer to the provided Quick Start Guide.

Use cases

This integration extracts dbt metadata using GraphQL calls and dbt API calls, then transforms the metadata and loads it to Collibra by Collibra API calls.

Elements in Scope

The integration is designed to retrieve the following elements:

  • SQL files
  • Database/Schema/Table/Column assets
  • KPI assets
  • DBT
  • GraphQL
  • Rest API calls
  • Reports
  • Data Quality Rules
  • Collibra Tags
  • System Asset

To receive support on this item, you can engage our Professional Services team or post any questions in the Data Citizens Community.

Media

More details

Release Notes
  • System asset and community bug fixes
  • Exception handlers have been added
Compatibility
  • Spring Boot Framework
  • Eclipse IDE
  • Collibra Data Intelligence Cloud
  • Collibra Data Intelligence On-Prem
Dependency
  • Java Runtime Environment 1.8
  • Spring Boot Integration Library
License and Usage Requirements
  • Collibra Data Intelligence Cloud

Release History

Version 1.1.5
December 1, 2022
Release Notes
  • Functionality added to create lineage directories by schema and dbt project
  • Functionality to create dynamic meta attributes added
  • Meta moved from tags to attribute
  • Request models simplified and HTTP method changed to POST
  • Collibra constants moved to application properties
  • Lineage directories bug fixes
Compatibility
  • Spring Boot Framework
  • Eclipse IDE
  • Collibra Data Intelligence Cloud
  • Collibra Data Intelligence On-Prem
Dependency
  • Java Runtime Environment 1.8
  • Spring Boot Integration Library
License and Usage Requirements
  • Collibra Data Intelligence Cloud
Version 1.1.4
November 22, 2022
Release Notes
  • Functionality to create or find domains added
  • Endpoints structure
  • Collibra community and domain structure
  • System asset and some application properties parameters moved to request body
Compatibility
  • Spring Boot Framework
  • Eclipse IDE
  • Collibra Data Intelligence Cloud
  • Collibra Data Intelligence On-Prem
Dependency
  • Java Runtime Environment 1.8
  • Spring Boot Integration Library
License and Usage Requirements
  • Collibra Data Intelligence Cloud
Version 1.1.3
October 31, 2022
Release Notes
  • Ingesting exposures have been added
  • Security updates
Compatibility
  • Spring Boot Framework
  • Eclipse IDE
  • Collibra Data Intelligence Cloud
  • Collibra Data Intelligence On-Prem
Dependency
  • Java Runtime Environment 1.8
  • Spring Boot Integration Library
License and Usage Requirements
  • Collibra Data Intelligence Cloud
Version 1.1.2
October 7, 2022
Release Notes
  • System assets have been added
  • Database assets naming and conventions changed
  • Obsolete method has been removed
Compatibility
  • Spring Boot Framework
  • Eclipse IDE
  • Collibra Data Intelligence Cloud
  • Collibra Data Intelligence On-Prem
Dependency
  • Java Runtime Environment 1.8
  • Spring Boot Integration Library
License and Usage Requirements
  • Collibra Data Intelligence Cloud
Version 1.1.0
September 29, 2022
Release Notes
  • Metrics ingestion and loading to Collibra as KPI assets
  • Spring Boot version updated
  • SpringFox replaced with Springdoc
  • Docker file improved
Compatibility
  • Spring Boot Framework
  • Eclipse IDE
  • Collibra Data Intelligence Cloud
  • Collibra Data Intelligence On-Prem
Dependency
  • Java Runtime Environment 1.8
  • Spring Boot Integration Library
License and Usage Requirements
  • Collibra Data Intelligence Cloud
Version 1.0.2
September 5, 2022
Release Notes
  • Technical Lineage database entities stitching functionality added.
  • Creating SQL files by project optional functionality added
  • JUnit 5 test classes added
  • Docker file has been added
Compatibility
  • Spring Boot Framework
  • Eclipse IDE
  • Collibra Data Intelligence Cloud
  • Collibra Data Intelligence On-Prem
Dependency
  • Java Runtime Environment 1.8
  • Spring Boot Integration Library
License and Usage Requirements
  • Collibra Catalog
Version 1.0.1
September 1, 2022
Release Notes
  • Lineage harvester methods removed from integration.
  • Extracted SQL timestamp name changed to model name.
  • Integration config resource bundle changed to externalize config class.
Compatibility
  • Spring Boot Framework
  • Eclipse IDE
  • Collibra Data Intelligence Cloud
  • Collibra Data Intelligence On-Prem
Dependency
  • Java Runtime Environment 1.8
  • Spring Boot Integration Library
License and Usage Requirements
  • Collibra Catalog
Version 1.0.0
August 1, 2022
Release Notes

Initial release.

Main features:

This Spring Boot integration extracts compiled SQLs and uploads SQL files using the Collibra lineage harvester.

Compatibility
  • Spring Boot Framework
  • Eclipse IDE
  • Collibra Data Intelligence Cloud
  • Collibra Data Intelligence On-Prem
Dependency
  • Java Runtime Environment 1.8
  • Spring Boot Integration Library
License and Usage Requirements
  • Collibra Catalog

See existing Q&A in the Data Citizens Community

Browse discussions with customers who also use this app. 

Start a New Topic in the Data Citizens Community

Collibra-hosted discussions will connect you to other customers who use this app.

The following terms shall apply to the extent you receive the source code to this offering.

Notwithstanding the terms of the Binary Code License Agreement under which this integration template is licensed, Collibra grants you, the Licensee, the right to access the source code to the integrated template in order to copy and modify said source code for Licensee’s internal use purposes and solely for the purpose of developing connections and/or integrations with Collibra products and services.

Solely with respect to this integration template, the term “Software,” as defined under the Binary Code License Agreement, shall include the source code version thereof. Except with respect to the foregoing, all remaining terms of the Binary Code License Agreement shall apply to the license of integration template hereunder.

Reviews

Rating
Leave a review