Knowledge Hub

This is a normal term for database modelers, but business users are not very familiar with it. Understand this and it will solve your 90% data problems. Don’t have good Data Model and you must deal with Chasm and Fan Traps.

Data Cardinality is about maintaining the level of details in a table. There are 2 types of Data Cardinality.

• High Cardinality: It is when there are no or less duplicates in a table. Tables with Pkey are mostly in Highest Cardinality mode.
• Low Cardinality: It is when there are high number of duplicates in a table. Tables with Fkey are mostly in Low Cardinality mode.

For example, Product table with details like Product ID, Product Name, and Product Manufactured Date etc. It can contain hundreds of products, but at the same time, a particular product cannot be repeated as it’s already in the table right. So, now we can say this table has High Cardinality. Referring to the same example if Product is there then one must sell it too. Sales of the products can be stored in a Sales table. Now, 1 type of product can be sold many times right. So, Sales tables can contain thousands of rows where hundreds of rows can be for 1 product means Product ID will be repeated in this Sales table so we can say this table is in Low Cardinality with respect to Products.

Another term, if data modelers are not aware and if data model doesn’t take care via proper Data Cardinality check and balance, data analysts are hit by Chasm and Fan Traps.

We know what Data Modelling is. We know what Data Cardinality is. Now, it’s time to understand Cartesian data which give nightmares to Finance department when they manually know the total is USD100k but when they run query, they see USD200 k 😊.

‘Cartesian Data is about writing a query on tables with ‘Many to Many (M:M)’ relationship.’

If there are two tables with ‘Many to Many’ relationship, it’s time for data modeler to introduce an intermediary table which can have unique records and then can have ‘One to Many’ relationship with previously used two tables.

If the Sales table is joined with Target tables directly based on Employee ID then surely there will Cartesian data as both Sales and Target tables can have more than 1 employee ID. But hang-on, even having intermediary table, if query is written wrongly then still data analyst can hit with Cartesian data know as Chasm Trap.

Almost every organization claims to recognize the importance of Data Governance, yet research and lived experience show that the majority still lack a functioning, enterprise-wide framework. Before the era of Big Data, governance was often ignored outside of highly regulated industries. But now, with a landscape filled with SQL and NoSQL databases, Data Warehouses, Data Lakes, Meshes, Lakehouses, Vaults, Hadoop clusters, Object Stores, and In-Memory platforms, governance is no longer optional, it is the operating system for how data is trusted and used.

The Purpose of Governance

Data Governance is not about bureaucracy or slowing down innovation. At its core, it is about balancing data as an asset with data as a liability. Done well, it creates clarity, consistency, and trust, the conditions for business value to emerge. Neglected, it leads directly to data swamps, failed projects, and regulatory exposure.

‘Data Quality is when we compare data against Business Rules.’

Organizations aspire to be data-driven, but the reality is that without quality, data becomes a liability rather than an asset. Decisions made on inaccurate, incomplete, or untimely data are not just ineffective, they can be damaging. Poor data quality has been repeatedly linked to regulatory fines, reputational loss, failed AI models, and massive operational costs. Simply put: bad data costs more than no data at all.

The Six Dimensions of Data Quality

• Accuracy – Data should reflect the real-world event or entity it represents. Accuracy depends on traceability through data lineage; without lineage, accuracy is always questionable.
• Completeness – Data must contain everything required to serve the business purpose. Completeness is not about “all data,” but about ensuring critical attributes exist to fulfill KPIs or regulatory needs.
• Consistency – Data values must not conflict across systems. If a customer’s birthdate appears in three formats across platforms, trust collapses.
• Validity – Data must adhere to business rules, constraints, and formats. For example, a date in DDMMYYYY or an amount rounded to four decimals.
• Uniqueness – Records must be free of duplication. Duplicate customer IDs or accounts lead to wasted marketing spend, skewed analytics, and customer frustration.
• Timeliness – Data must be available when promised. Missing overnight batches or delayed real-time feeds translate into broken business processes and lost opportunities.

Data Deduplication is mostly involved in ETL or ELT process.

‘Data Deduplication is about removing duplication in the data.’

There are two kinds of data which can be duplicated i.e., 1) Raw or Processed Data and 2) Absence of Golden Record

Types

1. Raw or Processed Data: With-in this type of data, there are many ways to have duplicate data but below are two main reasons why mostly it ended up generating duplicated data.
   1. Data entered from front-end applications like when system is so badly designed that it allows to enter duplicate data e.g., a customer is wrongly created twice, or a product sold once but system allows to enter it more than one time as human error. This data must be removed as part of the Data Quality process.
   2. Data transformed during ETL/ ELT process from source systems to Data Warehouse or Data Marts or Semantic Layers etc. This data also needs to be removed by Data Quality process.
2. Absence of Golden Record: Duplicates also occur when golden records are missing? What is Golden Record? Absence of Golden Records means, when there is more than one code for single entity e.g., consider an insurance company has a SAME TYPE OF POLICY with policy code as 001 in one system and 002 in second system and 003 in third system, now when data from all these three systems flows into a centralized location which can be a Data Warehouse or a Data Mart there will be transactions related to each of these policies but in reality all belong to SAME TYPE OF POLICY. When this data goes to the marketing department and they plan to send online campaigns, instead of sending 1 campaign for 1 type of policy, there will 3 campaigns designed for the same 1 type of policy. This scenario can be handled by a de-duplicated Golden Record with the help of Master Data Management (MDM) Application.

Data Observability is one of the most underrated terms, as its name says, it's about observing and providing analysis and analytics for decision making.

‘Data Observability comes under Metadata Management.’

Yes, Data Observability aims to focus on observing every data pipeline, data transformation, DDL, DML, data governance standards, security & privacy policies etc. across the organization. It creates a holistic view of everything, it assists to find out where the problem is and debugging steps assist to make future decisions, so the same problem don't occur in future.

Pillars of Data Observability

1. Freshness
2. Lineage
3. Schema
4. Volume
5. Distribution

Now the question is, observe but how? Do we or can we give access to one framework on all the frameworks in Enterprise? Of course, the Security, Risk, Compliance and Audit departments won't let it happen. But NOT-IF the access is requested only on tools for Metadata Management. Yes, Data Observability mainly access Enterprise Metadata Data.

We have been referring to three terms i.e., Master Data Management, Metadata Management and Data Lineage, in previous topics. Master Data and Metadata Management have already been covered. Now it’s time to decode Data Lineage which is THE most important FACTOR to save Data Lake from becoming a Data Swamp. Efficient Data Lineage can’t be built without Metadata in place.

I won’t go into details for Data Swamp, there is a separate topic on it. But any data showing in your BI Solution, and you are unable to track it back to its source means you don’t have Data Lineage and you are heading towards disaster as if not now but in future you will have no idea where the data is coming from and where it’s been used.

Data Lineage is the process of tagging all the touchpoints i.e., databases, tables, columns, business roles etc., from where data has been traveled from to BI solution or downstream systems. It also shows all the ETL(s) it went through, all the data enrichments, transformations, conversions, aggregations, and all the new measures created along the way. This helps businesses to trust the completeness and accuracy of the data for their decision-making. Data Lineage provides an audit trail of data lifecycle throughout the enterprise and assists users to understand UAM (User Access Metric).

Data Provenance and Data Lineage. Logically both do the same thing but at different levels of granularity.

Data Lineage is about knowing the Data Lifecycle but with respect to databases, tables, columns, business rule/ ETL rule etc. For example, data comes from which database, table, column and went into which database, table, column with what business rules incorporated during transformation process.

Data Provenance is at a higher level, it’s about Data Lifecycle with respect to inputs, entities, systems, and processes. For example, data comes in which format, from which source system, its related to which entity, what is the process used like real-time processing, batch processing or event based etc., any analytical modelling done, any data enrichment took place, any data cleaning, data quality rules, metadata management, master & reference data management etc., took place during the data lifecycle.

These two terms are used in Machine Learning for Data Mining and Data Science models. As we all know, within Machine Learning, there are Supervised and Unsupervised Learning methods.

Data Classification/Categorization=Supervised Learning

Data Clustering=Unsupervised Learnings

Both these terms are used for patterns identification before complex algorithms come into action.

Classification is done based on the uniqueness of the data where e.g., all animals are tagged separately whereas Clustering is done based on the characteristics e.g., 1) Animals with 2 legs 2) Animals with 4 legs.

• Data Classification uses labeled data whereas Data Clustering uses unlabeled data.
• The most popular Data Classification algorithms in data mining are the K-Nearest Neighbor and decision tree algorithms.
• The two common Data Clustering algorithms in data mining are K-means clustering and hierarchical clustering.
• Data Classification output is known.
• Data Clustering out is unknown.
• Training data is provided for Data Classification.
• No training data is provided for Data Clustering.

This is a very confusing topic as both Categorization and Classification are used interchangeably.

Both Categorization and Classification are Metadata and are used for Data Governance as well.

Categorization is based on the attributes and properties of entities in dataset whereas Classification is based on the business requirements.

As you can refer to the image, data has been categorized in passport, health record, financial record, job posting etc., and at the same time these categories are classified as Medium Risk.

‘Both these terms fall under Supervised Learning in Data Science domain.’

For example, there are grandparents, parent, children, cousins, servants, gardener, machinic, car painter etc., all these falls under categories but there can be further classifications i.e., family members and non-family members.

Categories: Grandparents, Parent, Children, Cousins, Gardener, Machinic, Car painters