Knowledge Hub

Data Deduplication is mostly involved in ETL or ELT process.

‘Data Deduplication is about removing duplication in the data.’

There are two kinds of data which can be duplicated i.e., 1) Raw or Processed Data and 2) Absence of Golden Record

Types

1. Raw or Processed Data: With-in this type of data, there are many ways to have duplicate data but below are two main reasons why mostly it ended up generating duplicated data.
   1. Data entered from front-end applications like when system is so badly designed that it allows to enter duplicate data e.g., a customer is wrongly created twice, or a product sold once but system allows to enter it more than one time as human error. This data must be removed as part of the Data Quality process.
   2. Data transformed during ETL/ ELT process from source systems to Data Warehouse or Data Marts or Semantic Layers etc. This data also needs to be removed by Data Quality process.
2. Absence of Golden Record: Duplicates also occur when golden records are missing? What is Golden Record? Absence of Golden Records means, when there is more than one code for single entity e.g., consider an insurance company has a SAME TYPE OF POLICY with policy code as 001 in one system and 002 in second system and 003 in third system, now when data from all these three systems flows into a centralized location which can be a Data Warehouse or a Data Mart there will be transactions related to each of these policies but in reality all belong to SAME TYPE OF POLICY. When this data goes to the marketing department and they plan to send online campaigns, instead of sending 1 campaign for 1 type of policy, there will 3 campaigns designed for the same 1 type of policy. This scenario can be handled by a de-duplicated Golden Record with the help of Master Data Management (MDM) Application.

Data Observability is one of the most underrated terms, as its name says, it's about observing and providing analysis and analytics for decision making.

‘Data Observability comes under Metadata Management.’

Yes, Data Observability aims to focus on observing every data pipeline, data transformation, DDL, DML, data governance standards, security & privacy policies etc. across the organization. It creates a holistic view of everything, it assists to find out where the problem is and debugging steps assist to make future decisions, so the same problem don't occur in future.

Pillars of Data Observability

1. Freshness
2. Lineage
3. Schema
4. Volume
5. Distribution

Now the question is, observe but how? Do we or can we give access to one framework on all the frameworks in Enterprise? Of course, the Security, Risk, Compliance and Audit departments won't let it happen. But NOT-IF the access is requested only on tools for Metadata Management. Yes, Data Observability mainly access Enterprise Metadata Data.

We have been referring to three terms i.e., Master Data Management, Metadata Management and Data Lineage, in previous topics. Master Data and Metadata Management have already been covered. Now it’s time to decode Data Lineage which is THE most important FACTOR to save Data Lake from becoming a Data Swamp. Efficient Data Lineage can’t be built without Metadata in place.

I won’t go into details for Data Swamp, there is a separate topic on it. But any data showing in your BI Solution, and you are unable to track it back to its source means you don’t have Data Lineage and you are heading towards disaster as if not now but in future you will have no idea where the data is coming from and where it’s been used.

Data Lineage is the process of tagging all the touchpoints i.e., databases, tables, columns, business roles etc., from where data has been traveled from to BI solution or downstream systems. It also shows all the ETL(s) it went through, all the data enrichments, transformations, conversions, aggregations, and all the new measures created along the way. This helps businesses to trust the completeness and accuracy of the data for their decision-making. Data Lineage provides an audit trail of data lifecycle throughout the enterprise and assists users to understand UAM (User Access Metric).

Data Provenance and Data Lineage. Logically both do the same thing but at different levels of granularity.

Data Lineage is about knowing the Data Lifecycle but with respect to databases, tables, columns, business rule/ ETL rule etc. For example, data comes from which database, table, column and went into which database, table, column with what business rules incorporated during transformation process.

Data Provenance is at a higher level, it’s about Data Lifecycle with respect to inputs, entities, systems, and processes. For example, data comes in which format, from which source system, its related to which entity, what is the process used like real-time processing, batch processing or event based etc., any analytical modelling done, any data enrichment took place, any data cleaning, data quality rules, metadata management, master & reference data management etc., took place during the data lifecycle.

These two terms are used in Machine Learning for Data Mining and Data Science models. As we all know, within Machine Learning, there are Supervised and Unsupervised Learning methods.

Data Classification/Categorization=Supervised Learning

Data Clustering=Unsupervised Learnings

Both these terms are used for patterns identification before complex algorithms come into action.

Classification is done based on the uniqueness of the data where e.g., all animals are tagged separately whereas Clustering is done based on the characteristics e.g., 1) Animals with 2 legs 2) Animals with 4 legs.

• Data Classification uses labeled data whereas Data Clustering uses unlabeled data.
• The most popular Data Classification algorithms in data mining are the K-Nearest Neighbor and decision tree algorithms.
• The two common Data Clustering algorithms in data mining are K-means clustering and hierarchical clustering.
• Data Classification output is known.
• Data Clustering out is unknown.
• Training data is provided for Data Classification.
• No training data is provided for Data Clustering.

Data Labelling is about tagging something which is not identifiable coming from source. The Labelling can be on known data and on unknown data as well.

Let’s try to understand the relationship among all these 3 terms i.e., Data Labelling, Data Classification and Data Clustering.

• Data Clustering group the data based on existing parameter and falls under Unsupervised Learning.
• Data Labelling is the process of tagging data based on some defined parameter.
• Data Classification group the data based on manually LABELLED data and falls under Supervised Learning.

Since the era of unstructured dataset, data is coming in many formats. There are specialized tools which can tag or label the unstructured data set so it can become input for Machine Learning (ML) Models. There are also certain libraries used in ML Models to label unstructured data set which can then become input of another ML Model for Supervised Learning.

Data Annotation and Data Labelling works together to prepare dataset for Machine Learning Models in Data Science practice.

‘Data Annotation is the technique of doing Data Labelling.’

Data Annotation is a complete process with many steps to curate, whereas Data Labelling is ONE of the many steps in it.

Data Labelling is about tagging the data for training e.g., machine learning models whereas Data Annotation is the technique for tagging the data so ML models can reuse it as well. Data Labelling tags are used by ML models for future pattern recognitions whereas Data Annotation is about recognizing the data via a process.

Just like Data Labelling, Data Annotation is also for supervised machine learning to prepare data as an input and to store it for future reference. Not to mention, there are STILL many organizations doing manual Data Labelling. Data Annotation is about the process so Data Labelling can be automated for accuracy, eliminating human intervention. Bad Data Labelling can impact the outcome of applications.

Data Hierarchy is a known term and most of us are aware of it but sometimes, it’s confused with Data Taxonomy & Data Ontology.

Data Taxonomy and Data Ontology are metadata for the purpose of Data Governance whereas Data Hierarchies presents actual data. A very well-known and famous example is the Organizational Chart where hierarchy starts from CEO and goes down to Database Administrator in data domain.

• ‘Data Taxonomies apply to identified objects or phenomena.’’
• ‘Data Ontologies apply to terms, concepts, or categories.’
• ‘Data Hierarchy is a chain of relationship in actually data.’

Family tree is another simple example.

Data can be grouped in a way where data exploration and representation become easy. For example, at database level, hierarchy can be Database 🡪 Files/ Tables 🡪 Records 🡪 Fields/ Columns 🡪 Bytes 🡪 Bits.

Same line Data Taxonomy, Data Hierarchy also have single parent and multiple child relationship among the rows of dataset.

Data Anonymization, Data Pseudonymization or Data De-identification all worked the same way and related to Data Privacy.

Data Anonymization is about removing or modifying personal information before sharing with others so those people whom the data about remains anonymized.

‘Data Anonymization is about hiding the identity of a person to be undetectable.’

For example, while implementing a data solution it is required to share data with development team, so before sharing one must make sure data is anonymized, persons details are not exposed to development team members e.g., name, contact number, address, credit card number etc.

Data Anonymization is normally enforced by Regulatory Bodies and not following can result in heavy penalties and a bad reputation for any organization.

Techniques [All these terms are covered separately]

• Data Masking
• Data Pseudonymization
• Data Generalization
• Data Swapping
• Data Perturbation
• Data Synthetic
• Data Shuffling
• Data Scrambling
• Data Encoding
• Data De-identification

Data Identification is the pre-requisite for Data Anonymization. How?

Data Identification is opposite to Data Anonymization, in other words, first one must identify data before conducting Data Anonymization.

Data Identification is about those set of details which assist in identifying an entity or individual. These details can be direct details like name, contact number, address, credit card number etc. or can be indirect details like unique identification number which assist to connect with other datasets to identify an entity or individual.

Data Identification is a very crucial exercise in data migration project where the process is to first conduct data discovery then data identification and then data migration can take place.

Data Identification can be done in all kinds of datasets i.e., structure, semi or unstructured but recommended practice is to do it in structured datasets.

Other Benefits