dbt to Collibra Integration
Overview
This Spring Boot integration extracts dbt metadata using GraphQL calls and dbt API calls. The integration then transforms the metadata and loads it to Collibra using Collibra API calls. This summarizes the simple flow of the ETL operations.
There are two main functionalities for this integration:
- Compile and extract SQLs from the dbt Cloud and create SQL files written to the physical local disk, ready for ingestion by the technical lineage harvester.
- Extract dbt project functionalities like models, exposures, metrics, tests, and metadata from the dbt Cloud, which are then transformed and loaded to Collibra.
In the case of models, the integration searches for an existing database domain named according to the format [ System asset name ] > [ Database ] > [ Schema ]. If the database exists, the integration updates it. Otherwise, it creates a new domain.
For dbt meta, the integration searches for attribute types by meta names given in the dbt project YML file. If there are none, the integration creates attribute types and loads assets with meta attributes. If you want to make this visible for the asset, you have to add characteristics from global assignments from the Collibra settings.
In the case of exposures, metrics, and tests, the integration looks for the name provided in the application properties file. If the domains exist, the integration updates them. Otherwise, new domains are created.
From the metrics, exposures, and tests GraphQL calls, the integration extracts metadata such as:
- Dimensions for relations (it is used for the columns),
- Filters (to apply to the underlying model for the metric),
- SQL expression (to calculate the metrics).
- Tests predicates
- Database assets from models
A CMA file is also included. Once this CMA file is imported within your Collibra instance, it will ensure that all the custom prerequisites will be generated automatically.
Alternative ways to try out this integration are via the supplied JAR file or Docker image with the compiled build version of the integration. These contain all the required dependencies and source code. After a minimal number of required changes as described in this document, users can run the integration using one simple command. Please refer to the provided Quick Start Guide.
Use cases
This integration extracts dbt metadata using GraphQL calls and dbt API calls, then transforms the metadata and loads it to Collibra by Collibra API calls.
Elements in Scope
The integration is designed to retrieve the following elements:
- SQL files
- Database/Schema/Table/Column assets
- KPI assets
- DBT
- GraphQL
- Rest API calls
- Reports
- Data Quality Rules
- Collibra Tags
- System Asset
To receive support on this item, you can engage our Professional Services team or post any questions in the Data Citizens Community.
Media
More details
Release Notes
- Swagger bug fixed
- Spring Boot version upgraded
Compatibility
- Spring Boot Framework
- Eclipse IDE
- Collibra Data Intelligence Cloud
- Collibra Data Intelligence On-Prem
Dependency
- Spring Boot Integration Library
- Java SDK 17
License and Usage Requirements
- Collibra Data Intelligence Cloud
Release History
Release Notes
- Continue on error functionality has been added
- Spring Boot version upgraded to 3.0.0
Compatibility
- Spring Boot Framework
- Eclipse IDE
- Collibra Data Intelligence Cloud
- Collibra Data Intelligence On-Prem
Dependency
- Spring Boot Integration Library
- Java SDK 17
License and Usage Requirements
- Collibra Data Intelligence Cloud
Release Notes
- dbt snapshots ingestion has been added
- Disable/Enable dbt Jobs flag has been added
- Creating extra directory bug fixed
Compatibility
- Spring Boot Framework
- Eclipse IDE
- Collibra Data Intelligence Cloud
- Collibra Data Intelligence On-Prem
Dependency
- Spring Boot Integration Library
- Java SDK 11
License and Usage Requirements
- Collibra Data Intelligence Cloud
Release Notes
- Java version migrated to Java 11
- Integration Library version upgraded regarding Java 11
Compatibility
- Spring Boot Framework
- Eclipse IDE
- Collibra Data Intelligence Cloud
- Collibra Data Intelligence On-Prem
Dependency
- Spring Boot Integration Library
- Java SDK 11
License and Usage Requirements
- Collibra Data Intelligence Cloud
Release Notes
- System asset and community bug fixes
- Exception handlers have been added
Compatibility
- Spring Boot Framework
- Eclipse IDE
- Collibra Data Intelligence Cloud
- Collibra Data Intelligence On-Prem
Dependency
- Java Runtime Environment 1.8
- Spring Boot Integration Library
License and Usage Requirements
- Collibra Data Intelligence Cloud
Release Notes
- Functionality added to create lineage directories by schema and dbt project
- Functionality to create dynamic meta attributes added
- Meta moved from tags to attribute
- Request models simplified and HTTP method changed to POST
- Collibra constants moved to application properties
- Lineage directories bug fixes
Compatibility
- Spring Boot Framework
- Eclipse IDE
- Collibra Data Intelligence Cloud
- Collibra Data Intelligence On-Prem
Dependency
- Java Runtime Environment 1.8
- Spring Boot Integration Library
License and Usage Requirements
- Collibra Data Intelligence Cloud
Release Notes
- Functionality to create or find domains added
- Endpoints structure
- Collibra community and domain structure
- System asset and some application properties parameters moved to request body
Compatibility
- Spring Boot Framework
- Eclipse IDE
- Collibra Data Intelligence Cloud
- Collibra Data Intelligence On-Prem
Dependency
- Java Runtime Environment 1.8
- Spring Boot Integration Library
License and Usage Requirements
- Collibra Data Intelligence Cloud
Release Notes
- Ingesting exposures have been added
- Security updates
Compatibility
- Spring Boot Framework
- Eclipse IDE
- Collibra Data Intelligence Cloud
- Collibra Data Intelligence On-Prem
Dependency
- Java Runtime Environment 1.8
- Spring Boot Integration Library
License and Usage Requirements
- Collibra Data Intelligence Cloud
Release Notes
- System assets have been added
- Database assets naming and conventions changed
- Obsolete method has been removed
Compatibility
- Spring Boot Framework
- Eclipse IDE
- Collibra Data Intelligence Cloud
- Collibra Data Intelligence On-Prem
Dependency
- Java Runtime Environment 1.8
- Spring Boot Integration Library
License and Usage Requirements
- Collibra Data Intelligence Cloud
Release Notes
- Metrics ingestion and loading to Collibra as KPI assets
- Spring Boot version updated
- SpringFox replaced with Springdoc
- Docker file improved
Compatibility
- Spring Boot Framework
- Eclipse IDE
- Collibra Data Intelligence Cloud
- Collibra Data Intelligence On-Prem
Dependency
- Java Runtime Environment 1.8
- Spring Boot Integration Library
License and Usage Requirements
- Collibra Data Intelligence Cloud
Release Notes
- Technical Lineage database entities stitching functionality added.
- Creating SQL files by project optional functionality added
- JUnit 5 test classes added
- Docker file has been added
Compatibility
- Spring Boot Framework
- Eclipse IDE
- Collibra Data Intelligence Cloud
- Collibra Data Intelligence On-Prem
Dependency
- Java Runtime Environment 1.8
- Spring Boot Integration Library
License and Usage Requirements
- Collibra Catalog
Release Notes
- Lineage harvester methods removed from integration.
- Extracted SQL timestamp name changed to model name.
- Integration config resource bundle changed to externalize config class.
Compatibility
- Spring Boot Framework
- Eclipse IDE
- Collibra Data Intelligence Cloud
- Collibra Data Intelligence On-Prem
Dependency
- Java Runtime Environment 1.8
- Spring Boot Integration Library
License and Usage Requirements
- Collibra Catalog
Release Notes
Initial release.
Main features:
This Spring Boot integration extracts compiled SQLs and uploads SQL files using the Collibra lineage harvester.
Compatibility
- Spring Boot Framework
- Eclipse IDE
- Collibra Data Intelligence Cloud
- Collibra Data Intelligence On-Prem
Dependency
- Java Runtime Environment 1.8
- Spring Boot Integration Library
License and Usage Requirements
- Collibra Catalog
See existing Q&A in the Data Citizens Community
Browse discussions with customers who also use this app.
Start a New Topic in the Data Citizens Community
Collibra-hosted discussions will connect you to other customers who use this app.
The following terms shall apply to the extent you receive the source code to this offering.
Notwithstanding the terms of the Binary Code License Agreement under which this integration template is licensed, Collibra grants you, the Licensee, the right to access the source code to the integrated template in order to copy and modify said source code for Licensee’s internal use purposes and solely for the purpose of developing connections and/or integrations with Collibra products and services.
Solely with respect to this integration template, the term “Software,” as defined under the Binary Code License Agreement, shall include the source code version thereof. Except with respect to the foregoing, all remaining terms of the Binary Code License Agreement shall apply to the license of integration template hereunder.
