Talend Data Quality Integration

Overview

Talend Data Quality offers an intuitive way to implement data quality best practices at scale, by profiling, cleaning, and masking data in real time, visualizing and tracking KPIs over time, and identifying and fixing data quality issues.

This Spring Boot integration gets data from a Talend Data Quality database, matches them and builds metrics with assets. Then, the transformed data are upserted as assets to a domain on the Collibra Platform.

Use cases

With this integration, you can:

• enjoy the freedom of ingesting metadata and data quality metrics from Talend directly into your Collibra instance
• be confident when planning and engaging with relevant business process owners, as the data center tracks changes in your Talend metadata.

Elements in Scope

The Talend data quality integration has only one HTTP post endpoint and this request sends two header parameters to the integration:

• X-Domain-ID, which indicates domain ID for matching assets
• X-Use-Mock-Data – a boolean parameter. If the parameter is false, the integration uses actual data from a database. Otherwise, it uses JSON mock data for testing purposes.

The main class is placed in the service layer:

public class IntegrationProcessorService

and the main method is placed in the class:

public String startIntegration(String domainId, boolean isMockData)

This method starts the integration process. Functionalities of the class are:

• read matching assets
• extract data
• import metrics to the Collibra platform.

For the transformation, logic is placed in the object

public class MetricTransformerComponent

This class retrieves matching assets from the service layer by filtering, which is done by comparing full names with metrics data attributes. If the metrics rule exists, then:

• the data is transformed
• the number of data rows that passed the rule is calculated
• the number of data rows that failed the rule is calculated
• the cost of a rule failing is measured (by aggregating transaction amounts, etc.)
• this finalizes the integration by importing metrics to the Collibra domain.

To receive support on this item, you can engage our Professional Services team or post any questions in the Data Citizens Community.