Knowledge Hub

Almost every organization claims to recognize the importance of Data Governance, yet research and lived experience show that the majority still lack a functioning, enterprise-wide framework. Before the era of Big Data, governance was often ignored outside of highly regulated industries. But now, with a landscape filled with SQL and NoSQL databases, Data Warehouses, Data Lakes, Meshes, Lakehouses, Vaults, Hadoop clusters, Object Stores, and In-Memory platforms, governance is no longer optional, it is the operating system for how data is trusted and used.

The Purpose of Governance

Data Governance is not about bureaucracy or slowing down innovation. At its core, it is about balancing data as an asset with data as a liability. Done well, it creates clarity, consistency, and trust, the conditions for business value to emerge. Neglected, it leads directly to data swamps, failed projects, and regulatory exposure.

‘Data Quality is when we compare data against Business Rules.’

Organizations aspire to be data-driven, but the reality is that without quality, data becomes a liability rather than an asset. Decisions made on inaccurate, incomplete, or untimely data are not just ineffective, they can be damaging. Poor data quality has been repeatedly linked to regulatory fines, reputational loss, failed AI models, and massive operational costs. Simply put: bad data costs more than no data at all.

The Six Dimensions of Data Quality

• Accuracy – Data should reflect the real-world event or entity it represents. Accuracy depends on traceability through data lineage; without lineage, accuracy is always questionable.
• Completeness – Data must contain everything required to serve the business purpose. Completeness is not about “all data,” but about ensuring critical attributes exist to fulfill KPIs or regulatory needs.
• Consistency – Data values must not conflict across systems. If a customer’s birthdate appears in three formats across platforms, trust collapses.
• Validity – Data must adhere to business rules, constraints, and formats. For example, a date in DDMMYYYY or an amount rounded to four decimals.
• Uniqueness – Records must be free of duplication. Duplicate customer IDs or accounts lead to wasted marketing spend, skewed analytics, and customer frustration.
• Timeliness – Data must be available when promised. Missing overnight batches or delayed real-time feeds translate into broken business processes and lost opportunities.

Data Deduplication is mostly involved in ETL or ELT process.

‘Data Deduplication is about removing duplication in the data.’

There are two kinds of data which can be duplicated i.e., 1) Raw or Processed Data and 2) Absence of Golden Record

Types

1. Raw or Processed Data: With-in this type of data, there are many ways to have duplicate data but below are two main reasons why mostly it ended up generating duplicated data.
   1. Data entered from front-end applications like when system is so badly designed that it allows to enter duplicate data e.g., a customer is wrongly created twice, or a product sold once but system allows to enter it more than one time as human error. This data must be removed as part of the Data Quality process.
   2. Data transformed during ETL/ ELT process from source systems to Data Warehouse or Data Marts or Semantic Layers etc. This data also needs to be removed by Data Quality process.
2. Absence of Golden Record: Duplicates also occur when golden records are missing? What is Golden Record? Absence of Golden Records means, when there is more than one code for single entity e.g., consider an insurance company has a SAME TYPE OF POLICY with policy code as 001 in one system and 002 in second system and 003 in third system, now when data from all these three systems flows into a centralized location which can be a Data Warehouse or a Data Mart there will be transactions related to each of these policies but in reality all belong to SAME TYPE OF POLICY. When this data goes to the marketing department and they plan to send online campaigns, instead of sending 1 campaign for 1 type of policy, there will 3 campaigns designed for the same 1 type of policy. This scenario can be handled by a de-duplicated Golden Record with the help of Master Data Management (MDM) Application.

Data Observability is one of the most underrated terms, as its name says, it's about observing and providing analysis and analytics for decision making.

‘Data Observability comes under Metadata Management.’

Yes, Data Observability aims to focus on observing every data pipeline, data transformation, DDL, DML, data governance standards, security & privacy policies etc. across the organization. It creates a holistic view of everything, it assists to find out where the problem is and debugging steps assist to make future decisions, so the same problem don't occur in future.

Pillars of Data Observability

1. Freshness
2. Lineage
3. Schema
4. Volume
5. Distribution

Now the question is, observe but how? Do we or can we give access to one framework on all the frameworks in Enterprise? Of course, the Security, Risk, Compliance and Audit departments won't let it happen. But NOT-IF the access is requested only on tools for Metadata Management. Yes, Data Observability mainly access Enterprise Metadata Data.

We have been referring to three terms i.e., Master Data Management, Metadata Management and Data Lineage, in previous topics. Master Data and Metadata Management have already been covered. Now it’s time to decode Data Lineage which is THE most important FACTOR to save Data Lake from becoming a Data Swamp. Efficient Data Lineage can’t be built without Metadata in place.

I won’t go into details for Data Swamp, there is a separate topic on it. But any data showing in your BI Solution, and you are unable to track it back to its source means you don’t have Data Lineage and you are heading towards disaster as if not now but in future you will have no idea where the data is coming from and where it’s been used.

Data Lineage is the process of tagging all the touchpoints i.e., databases, tables, columns, business roles etc., from where data has been traveled from to BI solution or downstream systems. It also shows all the ETL(s) it went through, all the data enrichments, transformations, conversions, aggregations, and all the new measures created along the way. This helps businesses to trust the completeness and accuracy of the data for their decision-making. Data Lineage provides an audit trail of data lifecycle throughout the enterprise and assists users to understand UAM (User Access Metric).

Data Provenance and Data Lineage. Logically both do the same thing but at different levels of granularity.

Data Lineage is about knowing the Data Lifecycle but with respect to databases, tables, columns, business rule/ ETL rule etc. For example, data comes from which database, table, column and went into which database, table, column with what business rules incorporated during transformation process.

Data Provenance is at a higher level, it’s about Data Lifecycle with respect to inputs, entities, systems, and processes. For example, data comes in which format, from which source system, its related to which entity, what is the process used like real-time processing, batch processing or event based etc., any analytical modelling done, any data enrichment took place, any data cleaning, data quality rules, metadata management, master & reference data management etc., took place during the data lifecycle.

These two terms are used in Machine Learning for Data Mining and Data Science models. As we all know, within Machine Learning, there are Supervised and Unsupervised Learning methods.

Data Classification/Categorization=Supervised Learning

Data Clustering=Unsupervised Learnings

Both these terms are used for patterns identification before complex algorithms come into action.

Classification is done based on the uniqueness of the data where e.g., all animals are tagged separately whereas Clustering is done based on the characteristics e.g., 1) Animals with 2 legs 2) Animals with 4 legs.

• Data Classification uses labeled data whereas Data Clustering uses unlabeled data.
• The most popular Data Classification algorithms in data mining are the K-Nearest Neighbor and decision tree algorithms.
• The two common Data Clustering algorithms in data mining are K-means clustering and hierarchical clustering.
• Data Classification output is known.
• Data Clustering out is unknown.
• Training data is provided for Data Classification.
• No training data is provided for Data Clustering.

This is a very confusing topic as both Categorization and Classification are used interchangeably.

Both Categorization and Classification are Metadata and are used for Data Governance as well.

Categorization is based on the attributes and properties of entities in dataset whereas Classification is based on the business requirements.

As you can refer to the image, data has been categorized in passport, health record, financial record, job posting etc., and at the same time these categories are classified as Medium Risk.

‘Both these terms fall under Supervised Learning in Data Science domain.’

For example, there are grandparents, parent, children, cousins, servants, gardener, machinic, car painter etc., all these falls under categories but there can be further classifications i.e., family members and non-family members.

Categories: Grandparents, Parent, Children, Cousins, Gardener, Machinic, Car painters

‘Data Segmentation is about dividing dataset into multiple sections based on some requirements.’

Segments are defined based on many parameters e.g., in a business, segments are defined by accessibility, actionable, profitable etc.

For example, when we go into a mall, there is a segment/section for Food Court where multiple categories of food are available with different classification like expensive or cheap food.

Steps to improve Data Segmentation

• Enrich your data to increase data quality.
• Plan and replan your data segmentation criteria(s)
• Improve real time segmentation.
• Design segment based on customer prioritization and feedback.
• Integrate with as many data elements.

In business, what you can do by having Data Segmentation

• You can create deals based on customer buying behavior.
• You can improve your products arrangement.
• You can plan your marketing strategy.
• You can analyze customer feedback.

Data Labelling is about tagging something which is not identifiable coming from source. The Labelling can be on known data and on unknown data as well.

Let’s try to understand the relationship among all these 3 terms i.e., Data Labelling, Data Classification and Data Clustering.

• Data Clustering group the data based on existing parameter and falls under Unsupervised Learning.
• Data Labelling is the process of tagging data based on some defined parameter.
• Data Classification group the data based on manually LABELLED data and falls under Supervised Learning.

Since the era of unstructured dataset, data is coming in many formats. There are specialized tools which can tag or label the unstructured data set so it can become input for Machine Learning (ML) Models. There are also certain libraries used in ML Models to label unstructured data set which can then become input of another ML Model for Supervised Learning.