Fivetran to Collibra Integration (Python)
Overview
Fivetran is a SAAS-based data integration service that helps you centralize data from disparate sources which you can manage directly from your browser. You can think about Fivetran as a data pipeline that allows you to connect, extract, and load data (and even perform post-transformation) into your data destination using supported connectors.
This Python integration extracts metadata from Fivetran and imports them to the Collibra Platform. The Fivetran entities that the integration will import as assets include:
- Fivetran Groups (as a community),
- Fivetran Destinations,
- Fivetran Connectors,
- Fivetran Tasks,
- Fivetran Warnings.
In addition to this, the integration will create relationships/links to the matching Catalog assets (Database, Schema, Table) which are ingested separately in Collibra using the Data Catalog.
Use Cases
- As a Data Engineer or Data Architect, I want to import Fivetran Destinations, Connectors, Tasks, and Warnings so I can visualize these Fivetran entities in Collibra and how they’re related to each other.
- As a Data Engineer or Data Architect, I want to link Fivetran Destinations, Connectors, Tasks, and Warnings to the ingested Catalog data assets so I can see an end-to-end lineage of the Connector data assets and the Destination data assets.
Note: This is a python application that can run as a job or REST web service to create templates and ingest Fivetran data to Collibra. More details in the documentation.
To receive support on this item, you can engage our Professional Services team or post any questions in the Data Citizens Community.
Media
More details
Release Notes
-
Limited to only Edge ingested Catalog Metadata due to complexity
-
Custom Technical Lineage or column level lineage between Fivetran Connector source metadata and Fivetran Destination target metadata
Compatibility
- Fivetran API V1
- Python
- Collibra Data Intelligence Cloud
Dependency
- Python 3.9+
- Fivetran API V1
License and Usage Requirements
Release History
Release Notes
- Fix issue with creating the relation between the Connectors and Columns ([Fivetran Connector] loads [Column])
Compatibility
- Fivetran API V1
- Python
- Collibra Data Intelligence Cloud
- Collibra Data Intelligence On-Prem
Dependency
- Python 3.9+
- Fivetran API V1
License and Usage Requirements
Release Notes
-
Added option to extract configurations from an external *.ini file
-
Added option to extract configurations from a Vault or Secrets manager without making changes to the template
-
Added optional command parameters for each options above (make sure that you review this)
-
This version should be backward compatible with v2.0.2 when the default options is used
-
Updated the collibra python packages to support JWT
-
ACCESS_TOKEN was added as one of the configurations to send Bearer token (JWT) when making API calls to Collibra
-
At the time of this writing the latest version on PyPi is >=2.0.0
-
Compatibility
- Fivetran API V1
- Python
- Collibra Data Intelligence Cloud
- Collibra Data Intelligence On-Prem
Dependency
- Fivetran API V1
- Python 3.9+
License and Usage Requirements
Release Notes
- Fixed some errors with mapping to Fivetran API v1 e.g. from “name“ of Connector to “schema“.
- Updated CMA file.
Compatibility
- Fivetran API V1
- Python
- Collibra Data Intelligence Cloud
- Collibra Data Intelligence On-Prem
Dependency
- Fivetran API V1
- Python 3.9+
License and Usage Requirements
Release Notes
- Updated to Fivetran API v1 (released version)
- Added relation between Fivetran Connector and Column Replaced the folder collibra_api_client by packages pulled from PyPi. See the requirements.txt
- Add notes about the System Location attribute
Compatibility
- Fivetran API V1
- Python
- Collibra Data Intelligence On-Prem
- Collibra Data Intelligence Cloud
Dependency
- Fivetran API V1
- Python 3.9+
License and Usage Requirements
Release Notes
-
Update operating model based on initial feedback of v1.0.0
-
Revised relationships and removed the fivetran group mapping file
-
Removed Column to Column lineage
-
Removed links between Connector and Source Schema
-
-
Utilize Fivetran API v2
Compatibility
- Python
- Fivetran API V2
- Collibra Data Intelligence Cloud
- Collibra Data Intelligence On-Prem
Dependency
- Fivetran API v2
- Python 3.9+
License and Usage Requirements
Release Notes
Python Application with the following main features:
- Represent Fivetran Groups, Destinations, Connectors as assets in Collibra.
- Links Destinations/Connectors to Catalog assets.
- Create column level lineages between Catalog assets linked to Destination and Connectors.
Compatibility
- Python
- Fivetran API
- Collibra Data Intelligence Cloud
- Collibra Data Intelligence On-Prem
Dependency
- Collibra API v2
- Python3.9+
License and Usage Requirements
See existing Q&A in the Data Citizens Community
Browse discussions with customers who also use this app.
Start a New Topic in the Data Citizens Community
Collibra-hosted discussions will connect you to other customers who use this app.
The following terms shall apply to the extent you receive the source code to this offering.
Notwithstanding the terms of the Binary Code License Agreement under which this integration template is licensed, Collibra grants you, the Licensee, the right to access the source code to the integrated template in order to copy and modify said source code for Licensee’s internal use purposes and solely for the purpose of developing connections and/or integrations with Collibra products and services.
Solely with respect to this integration template, the term “Software,” as defined under the Binary Code License Agreement, shall include the source code version thereof. Except with respect to the foregoing, all remaining terms of the Binary Code License Agreement shall apply to the license of integration template hereunder.
