Data Lineage

We have been referring to three terms i.e., Master Data Management, Metadata Management and Data Lineage, in previous topics. Master Data and Metadata Management have already been covered. Now it’s time to decode Data Lineage which is THE most important FACTOR to save Data Lake from becoming a Data Swamp. Efficient Data Lineage can’t be built without Metadata in place.

I won’t go into details for Data Swamp, there is a separate topic on it. But any data showing in your BI Solution, and you are unable to track it back to its source means you don’t have Data Lineage and you are heading towards disaster as if not now but in future you will have no idea where the data is coming from and where it’s been used.

Data Lineage is the process of tagging all the touchpoints i.e., databases, tables, columns, business roles etc., from where data has been traveled from to BI solution or downstream systems. It also shows all the ETL(s) it went through, all the data enrichments, transformations, conversions, aggregations, and all the new measures created along the way. This helps businesses to trust the completeness and accuracy of the data for their decision-making. Data Lineage provides an audit trail of data lifecycle throughout the enterprise and assists users to understand UAM (User Access Metric).

Proper Data Lineage process in place can bring down 80% of the efforts required to resolve errors. It also eliminates the human error factor while doing impact analysis of any error or any new CR required on existing solutions. It greatly helps in finding the root cause of error in GUI mode and assists in resolution with minimal chance of missing any elements.

In a nutshell, Data Lineage helps the BAU Support team to find out the data tracing from source till the endpoint for R&D.

‘Data Lineage is the low-level details of Data Provenance.’