Data Provenance

Data Provenance and Data Lineage. Logically both do the same thing but at different levels of granularity.

Data Lineage is about knowing the Data Lifecycle but with respect to databases, tables, columns, business rule/ ETL rule etc. For example, data comes from which database, table, column and went into which database, table, column with what business rules incorporated during transformation process.

Data Provenance is at a higher level, it’s about Data Lifecycle with respect to inputs, entities, systems, and processes. For example, data comes in which format, from which source system, its related to which entity, what is the process used like real-time processing, batch processing or event based etc., any analytical modelling done, any data enrichment took place, any data cleaning, data quality rules, metadata management, master & reference data management etc., took place during the data lifecycle.

‘Data Lineage is mostly used by IT folks whereas Data Provenance is used by Business Users or Management.’