Azure Data Factory to Collibra Integration
Overview
Azure Data Factory is a cloud-based ETL and data integration service with capabilities to create and schedule data-driven workflows from various data stores. Data can be transformed visually with data flows or though services like Azure Databricks, Azure HDInsight Hadoop, and Azure SQL Database.
This Spring Boot integration retrieves metadata from Azure Data Factory, transforms, and upserts it to a Collibra Platform instance as assets.
Use Cases
The Azure Data Factory integration serves the following use cases (amongst others):
- Increased trust and data citizen engagement around data streams that pass through, are collected by or produced in Azure Data Factory.
- Enhanced data lineage diagrams, data dictionaries and business glossaries. In many cases data is captured, transformed and sourced from Azure Data Factory with little documentation.
- The data center can track changes in Azure Data Factory metadata in order to plan and engage with relevant business process owners accordingly.
Elements in Scope
The integration is designed to retrieve the following metadata:
- Pipeline
- Parameter
- Variable
- Activity
- User Property
To receive support on this item, you can engage our Professional Services team or post any questions in the Data Citizens Community.
Media
More details
Release Notes
- Updated the Spring Boot Starter Parent version to 2.7.5
- Updated the Collibra Integration Library version to 1.1.10
- Add Docker file
- Add CMA file
Compatibility
- Spring Boot Framework
- Eclipse IDE
- Collibra Data Intelligence Cloud
- Collibra Data Intelligence On-Prem
Dependency
- Java Runtime Environment 1.8
- Spring Boot Integration Library
- Microsoft Data Factory API version 2018-06-01
License and Usage Requirements
Release History
Release Notes
- Updated the Spring Boot Starter Parent version to 2.5.12 (CVE-2022-22965).
- Updated the Collibra Integration Library version to 1.1.5.
Compatibility
- Spring Boot Framework
- Collibra Data Intelligence Cloud
- Collibra Data Intelligence On-Prem
Dependency
- Java Runtime Environment 1.8
- Spring Boot Integration Library
- Microsoft Data Factory API version 2018-06-01
License and Usage Requirements
Release Notes
Updated the Spring Boot Integration Library dependency version in the pom.xml file to v1.1.3 that supports the latest Collibra Platform versions (v2022.01+).
Compatibility
- Spring Boot Framework
- Collibra Data Intelligence Cloud
- Collibra Data Intelligence On-Prem
Dependency
- Java Runtime Environment 1.8
- Spring Boot Integration Library
- Microsoft Data Factory API version 2018-06-01
License and Usage Requirements
Release Notes
Updated Log4J Apache from v2.16.0 to v2.17.0
Compatibility
- Spring Boot Framework
- Collibra Data Intelligence Cloud
- Collibra Data Intelligence On-Prem
Dependency
- Java Runtime Environment 1.8
- Spring Boot Integration Library
- Microsoft Data Factory API version 2018-06-01
License and Usage Requirements
Release Notes
Updated logger log4j2 dependency to Apache log4j2 version 2.16.0.
Compatibility
- Spring Boot Framework
- Collibra Data Intelligence Cloud
- Collibra Data Intelligence On-Prem
Dependency
- Java Runtime Environment 1.8
- Collibra Platform v2021+
- Spring Boot Integration Library
- Microsoft Data Factory API version 2018-06-01
License and Usage Requirements
Release Notes
Initial release:
Spring Boot integration that retrieves metadata from Azure Data Factory using the API, transforms it and upserts it as assets on the Collibra Platform instance.
Note: To avoid some potential conflicts with sbi-azure-data-factory integration that was developed using SDK, it has been decided that first version of this integration will be 1.1.0.
Compatibility
- Spring Boot Framework
- Eclipse IDE
- Collibra Data Intelligence Cloud
- Collibra Data Intelligence On-Prem
Dependency
- Java Runtime Environment 1.8
- Collibra Platform v2021+
- Spring Boot Integration Library
- Microsoft Data Factory API version 2018-06-01
License and Usage Requirements
Release Notes
Initial release.
Main features:
- Spring Boot integration that retrieves metadata from Azure Data Factory, transforms it and upserts it as assets on the Collibra Platform instance
- Logic for marking old assets as obsolete
Compatibility
- Spring Boot Framework
- Eclipse IDE
- Collibra Data Intelligence Cloud
Dependency
- Java Runtime Environment 8
- Spring Boot Framework
- Collibra Integration library
- Java Development Kit v1.8
- Azure Subscription
- Azure Resource Manager DataFactory client library for Java Version 1.0.0-beta.5
License and Usage Requirements
See existing Q&A in the Data Citizens Community
Browse discussions with customers who also use this app.
Start a New Topic in the Data Citizens Community
Collibra-hosted discussions will connect you to other customers who use this app.
The following terms shall apply to the extent you receive the source code to this offering: Notwithstanding the terms of the Binary Code License Agreement under which this integration template is licensed, Collibra grants you, the Licensee, the right to access the source code to the integrated template in order to copy and modify said source code for Licensee’s internal use purposes and solely for the purpose of developing connections and/or integrations with Collibra products and services.
Solely with respect to this integration template, the term “Software,” as defined under the Binary Code License Agreement, shall include the source code version thereof. Except with respect to the foregoing, all remaining terms of the Binary Code License Agreement shall apply to the license of integration template hereunder.
