Collibra Data Product to Databricks Metric View

This workflow, implemented as a Flowable BPMN workflow, allows authorised Collibra users to generate and deploy a Databricks Metric View from a data product asset using the latest Semantic Context Specification API.

At its core, the semantic context specification it is a Key-Based YAML configuration language that users author (visually or in code) to describe how to traverse Collibra’s graph and what to extract at each step. The engine compiles this configuration into optimized queries against the Collibra Platform API, fetches the data, and then post-processes the result into the target format.

Limitations

When starting fresh:
– the model doesn’t differentiate source from join tables
– joined columns are determined by a simplistic name similarity algorithm
– each source table ends as a metric view, the blueprint API produces only one

When resuming work:
– no option to add/change joins or define explicit table relationships expressions
– often fails when moving columns between dimension (fields) and measure lists

In addition:
– Databricks is quite picky when choosing names and display names for columns
– Expressions depend on the chosen dimension and measure name, which in turn:

• makes it hard to retain changes made in the metric view