Bluemetrix Data Manager (BDM)

BDM is a full-service Spark based ETL solution which integrates with Collibra Data Platform to create, capture and maintain the data governance state of all data assets processed in the pipeline. By default, the following functionality is available:

• Capture and creation of lineage data
• Capture and creation of all technical meta-data
• Capture and creation of all business meta-data
• Capture and creation of GDPR compliance data
• Capture and creation of other compliance data as required
• Capture and creation of Tags

BDM enforces all Tag based, and Standard based Data Policies that exist in the Collibra Data Platform. It does this in the following manner:

• For all data sources in the pipeline, check for any Tags and Standards that govern the data source
• For each Tag and Standard found, determine if a corresponding rule exists in BDM, e.g., all data that has a PII Tag has to be tokenized on Write
• Before the pipeline executes a Write, apply all Tag or Standard based rules to the pipeline, e.g., for all data assets that are tagged PII, automatically apply a Tokenization function to the data before it is written to its destination.

BDM works with and has connectors for all major data sources (Oracle, Teradata, Snowflake, DB2, etc.) and types (Databases, CSV, JSON, EBCDIC, Kafka, etc.). If a connector is not available and is needed for a custom data source, we will create it as required.

BDM has over 260 pre-programmed Spark functions available as standard, and custom functionality can be created by Bluemetrix or the customer using UDF functionality.

BDM also supports SQL coding for advanced users who create or use existing SQL code in their pipelines.

BDM also provides Tokenization functionality using Format Preserving Encryption which is FIPS compliant.

The BDM Tokenization functionality is also available as an External Function for Snowflake in the Azure cloud. Support for external functionality for AWS and GCP is currently being worked on and will be available for release shortly.