Unsupported Screen Size: The viewport size is too small for the theme to render properly.

Talend Data Quality Integration

Published by: Collibra Marketplace
Latest version: 1.0.5
Released: June 24, 2022
Package Documentation
Community Offering

Your use of Community Offerings is subject to the Collibra Marketplace License Agreement. Read more.

Overview

For this specific integration (and all other Custom Integrations listed on the Collibra Marketplace), please read the following disclaimer:

  • This integration is a template that has been developed in cooperation with a few select clients based on their custom use cases and business needs.
  • While all effort has been made to encompass a range of typical usage scenarios, specific needs beyond this may require chargeable template customization.
  • With this in mind, we have made sure that the template is available as source code and readily modifiable to suit the client's particular use case.

Overview

Talend Data Quality offers an intuitive way to implement data quality best practices at scale, by profiling, cleaning, and masking data in real time, visualizing and tracking KPIs over time, and identifying and fixing data quality issues.

This Spring Boot integration gets data from a Talend Data Quality database, matches them and builds metrics with assets. Then, the transformed data are upserted as assets to a domain on the Collibra Platform.

Use cases

With this integration, you can:

  • enjoy the freedom of ingesting metadata and data quality metrics from Talend directly into your Collibra instance
  • be confident when planning and engaging with relevant business process owners, as the data center tracks changes in your Talend metadata.

Elements in Scope

The Talend data quality integration has only one HTTP post endpoint and this request sends two header parameters to the integration:

  • X-Domain-ID, which indicates domain ID for matching assets
  • X-Use-Mock-Data – a boolean parameter. If the parameter is false, the integration uses actual data from a database. Otherwise, it uses JSON mock data for testing purposes.

The main class is placed in the service layer:

public class IntegrationProcessorService

and the main method is placed in the class:

public String startIntegration(String domainId, boolean isMockData)

This method starts the integration process. Functionalities of the class are:

  • read matching assets
  • extract data
  • import metrics to the Collibra platform.

For the transformation, logic is placed in the object

public class MetricTransformerComponent

This class retrieves matching assets from the service layer by filtering, which is done by comparing full names with metrics data attributes. If the metrics rule exists, then:

  • the data is transformed
  • the number of data rows that passed the rule is calculated
  • the number of data rows that failed the rule is calculated
  • the cost of a rule failing is measured (by aggregating transaction amounts, etc.)
  • this finalizes the integration by importing metrics to the Collibra domain.

To receive support on this item, you can engage our Professional Services team or post any questions in the Data Citizens Community.

Media

More details

Release Notes
  • Updated the Spring Boot Starter Parent version to 2.5.12
  • Updated the Collibra Integration Library version to 1.1.5
Compatibility
  • Spring Boot Framework
  • Eclipse IDE
  • Collibra Cloud
  • Collibra 5.7.5 and newer
Dependency
  • Java Runtime Environment 1.8
  • Spring Boot Integration Library
License and Usage Requirements
  • Collibra Catalog
  • Talend

Release History

Version 1.0.4
March 21, 2022
Release Notes

Updated the Spring Boot Integration Library dependency version in the pom.xml file to v1.1.3 that supports the latest Collibra Platform versions (v2022.01+).

Compatibility
  • Spring Boot Framework
  • Eclipse IDE
  • Collibra Cloud
  • Collibra 5.7.5 and newer
Dependency
  • Java Runtime Environment 1.8
  • Spring Boot Integration Library
License and Usage Requirements
  • Collibra Catalog
  • Talend
Version 1.0.3
December 22, 2021
Release Notes

Updated the Log4j version from 2.16 to 2.17 due to vulnerabilities.

Compatibility
  • Spring Boot Framework
  • Eclipse IDE
  • Collibra Cloud
  • Collibra 5.7.5 and newer
Dependency
  • Java Runtime Environment 1.8
  • Spring Boot Integration Library
License and Usage Requirements
  • Collibra Catalog
  • Talend
Version 1.0.2
December 16, 2021
Release Notes

Updated logger log4j2 dependency to Apache log4j2 version 2.16.0.

Compatibility
  • Spring Boot Framework
  • Eclipse IDE
  • Collibra Cloud
  • Collibra 5.7.5 and newer
Dependency
  • Java Runtime Environment 1.8
  • Spring Boot Integration Library
License and Usage Requirements
  • Collibra Catalog
  • Talend
Version 1.0.1
August 24, 2021
Release Notes
  • Added the log4j2.xml file
  • Added note regarding using an external KeyStore file
Compatibility
  • Spring Boot Framework
  • Eclipse IDE
  • Collibra Cloud
  • Collibra 5.7.5 and newer
Dependency
  • Java Runtime Environment 1.8
  • Spring Boot Integration Library
License and Usage Requirements
  • Talend
  • Collibra Catalog
Version 1.0.0
August 5, 2021
Release Notes

– Initial release:

Spring Boot integration that gets data from a Talend data quality database, matches it and builds metrics with assets, and upserts the transformed data as assets to a domain on the Collibra Platform.

Compatibility
  • Spring Boot Framework 2.5.0
  • Eclipse IDE
  • Collibra Cloud
  • Collibra 5.7.5 and newer
Dependency
  • Java Runtime Environment 8
  • Spring Boot Framework 2.5.0
License and Usage Requirements
  • Talend

To receive support on this item, you can engage our Professional Services team or post any questions in the Data Citizens Community.

The following terms shall apply to the extent you receive the source code to this offering.

Notwithstanding the terms of the Binary Code License Agreement under which this integration template is licensed, Collibra grants you, the Licensee, the right to access the source code to the integrated template in order to copy and modify said source code for Licensee’s internal use purposes and solely for the purpose of developing connections and/or integrations with Collibra products and services.

Solely with respect to this integration template, the term “Software,” as defined under the Binary Code License Agreement, shall include the source code version thereof. Except with respect to the foregoing, all remaining terms of the Binary Code License Agreement shall apply to the license of integration template hereunder.

Reviews

Rating
Leave a review