Knowledge Hub

Data Deduplication is mostly involved in ETL or ELT process.

‘Data Deduplication is about removing duplication in the data.’

There are two kinds of data which can be duplicated i.e., 1) Raw or Processed Data and 2) Absence of Golden Record

Types

1. Raw or Processed Data: With-in this type of data, there are many ways to have duplicate data but below are two main reasons why mostly it ended up generating duplicated data.
   1. Data entered from front-end applications like when system is so badly designed that it allows to enter duplicate data e.g., a customer is wrongly created twice, or a product sold once but system allows to enter it more than one time as human error. This data must be removed as part of the Data Quality process.
   2. Data transformed during ETL/ ELT process from source systems to Data Warehouse or Data Marts or Semantic Layers etc. This data also needs to be removed by Data Quality process.
2. Absence of Golden Record: Duplicates also occur when golden records are missing? What is Golden Record? Absence of Golden Records means, when there is more than one code for single entity e.g., consider an insurance company has a SAME TYPE OF POLICY with policy code as 001 in one system and 002 in second system and 003 in third system, now when data from all these three systems flows into a centralized location which can be a Data Warehouse or a Data Mart there will be transactions related to each of these policies but in reality all belong to SAME TYPE OF POLICY. When this data goes to the marketing department and they plan to send online campaigns, instead of sending 1 campaign for 1 type of policy, there will 3 campaigns designed for the same 1 type of policy. This scenario can be handled by a de-duplicated Golden Record with the help of Master Data Management (MDM) Application.

Data Observability is one of the most underrated terms, as its name says, it's about observing and providing analysis and analytics for decision making.

‘Data Observability comes under Metadata Management.’

Yes, Data Observability aims to focus on observing every data pipeline, data transformation, DDL, DML, data governance standards, security & privacy policies etc. across the organization. It creates a holistic view of everything, it assists to find out where the problem is and debugging steps assist to make future decisions, so the same problem don't occur in future.

Pillars of Data Observability

1. Freshness
2. Lineage
3. Schema
4. Volume
5. Distribution

Now the question is, observe but how? Do we or can we give access to one framework on all the frameworks in Enterprise? Of course, the Security, Risk, Compliance and Audit departments won't let it happen. But NOT-IF the access is requested only on tools for Metadata Management. Yes, Data Observability mainly access Enterprise Metadata Data.

Data Labelling is about tagging something which is not identifiable coming from source. The Labelling can be on known data and on unknown data as well.

Let’s try to understand the relationship among all these 3 terms i.e., Data Labelling, Data Classification and Data Clustering.

• Data Clustering group the data based on existing parameter and falls under Unsupervised Learning.
• Data Labelling is the process of tagging data based on some defined parameter.
• Data Classification group the data based on manually LABELLED data and falls under Supervised Learning.

Since the era of unstructured dataset, data is coming in many formats. There are specialized tools which can tag or label the unstructured data set so it can become input for Machine Learning (ML) Models. There are also certain libraries used in ML Models to label unstructured data set which can then become input of another ML Model for Supervised Learning.

Data Identification is the pre-requisite for Data Anonymization. How?

Data Identification is opposite to Data Anonymization, in other words, first one must identify data before conducting Data Anonymization.

Data Identification is about those set of details which assist in identifying an entity or individual. These details can be direct details like name, contact number, address, credit card number etc. or can be indirect details like unique identification number which assist to connect with other datasets to identify an entity or individual.

Data Identification is a very crucial exercise in data migration project where the process is to first conduct data discovery then data identification and then data migration can take place.

Data Identification can be done in all kinds of datasets i.e., structure, semi or unstructured but recommended practice is to do it in structured datasets.

Other Benefits

Data Generalization/ Blurring is normally confused with Data Anonymization which is about removing or modifying personal information so the identity of a person can be undetectable. Data Generalization aim is same but using with different approach.

‘Data Generalization/ Blurring and Specialization is something like Zoom-Out vs Zoom-In or Precise vs Less-Precise.’

One of the main requirements for Data Generalization comes into play when you must maintain Data Sovereignty (explain in a separate topic) but keep the data originality. It is used to cater for data privacy.

‘Data Generalization summarized data in such a way that the usefulness of the data remains intact.’

Data Specialization is the right opposite of Data Generalization as it divides entity into multiple entities to further make sense of the data.

Example of Data Generalization: Referring to the image, Students and Patients both have Name and Mobile numbers, we can have summarized entity like Person which can host both Students and Patients details i.e., Roll_No and PID numbers. This way, there will be records of all the students and patients, but no one will be able to identify any student or patient.

Data Parsing and Formatting sometime works interchangeably and mixed with Data Conversion and Data Transformation which is not correct.

Parsing term comes from Latin part (orationis), meaning part (of speech) [Wikipedia].

We all must have heard about parser. Yes, Data Parsing is about rearranging or formatting the data in such a way that it’s understandable or usable. In Data Parsing, data is not anonymized neither does it changes its meaning rather it changes for the sake of its understandable.

Referring to the image, we can see data in columns are rearranged and now it can make sense. If we pull it in a reports or dashboard, the decisions can be taken, of course not on un-parsed data.

In the era of social media, most of online portals gives APIs to allow third parties to extract data in a readable format but those who doesn’t provide APIs, we use Web Scrapping (explained in separate topic) to crawl the websites, online portals etc., to extract details by running different type of parsing approaches.

Most people confuse Quality and Data Integrity with each other.

The term Data Integrity came from a security point of view, but it came from sharing data in a more effective and easier way, so in a way, it is more than just security. It has multiple components and Data Quality is one of those. In other words, these are not two different terms, nor both compete, rather Data Quality supplements Data Integrity along with other components.

Achieving a higher % age of Data Integrity gives confidence to the business users and management to trust data for daily decision making. Losing Data Integrity means a higher possibility of wrong data, wrong data means wrong decisions, and wrong decisions mean business lose margin or go into loss. Security’s point of view is to make sure the results shared with Public Stakeholders or Regulatory Bodies are correct and not fake. That's the reason there are internal and external audit teams evaluating the Data Integrity of organizations. Based on audit results, more investors invest, more buyers buy, and the company flourishes.

All these terms like Data Integrity, Data Quality, Data Scrubbing, Data Cleansing and Data Cleaning are very interrelated.

• Cleaning: You eat something, and you get something on your face, you clean your face. You are painting something, and you get splashes of paint on your face, you clean your face. You are walking and you feel the dust, you wash and clean your face.
  SAME WAY, data cleaning can be done at a very high level by Business Users themselves as it requires little effort e.g., export data in excel, clean it, and send it downstream.
• Cleansing: You went on safari or played a football match on muddy ground, now you must use some face cream to cleanse your face plus you have to spend 10-15 minutes. Then it is better to ask someone or ideally go to the parlor for good face cleansing.
  SAME WAY, data cleansing can be done by business users themselves but ideally IT SME(s) can write SQL queries at the database level to cleanse data so next time when you extract, you get the right cleansed data.
• Scrubbing: This is the best level to achieve the highest Data Quality. Coming back to the same example, you need to attend a wedding or birthday party or any kind of valuable event, now you will go to a proper parlor, you will find the best worker, you will select the best brand of creams (based on your pocket vs the event you are attending) to scrub your face skin. You will make sure your face has been properly cleaned, cleansed, and scrubbed, and you will make sure you look ready for the event.
  SAME WAY, data scrubbing is to make sure your data is near to perfect, you will find tools that can do this for you, you will find good resources for this task, you will make sure the findings are implemented from Source to Downstream systems.

Data Wrangling or Data Mangling or Data Munging are used interchangeably and do the same function.

Data Wrangling is something that every ETL developer, Data Engineer, Data Analyst etc. do daily. It is something that is done before Data Mining, Data Science, Business Intelligence or Business Analytics.

In simple words, Data Wrangling is to prepare data for processing. Note, prepare data for processing, not process data. Yes, you heard it correctly, sometimes Data Wrangling techniques are used for permanent data fixes and sometimes to prepare data for specific use cases.

Let's take an example, refer to the tables above, fixing and streamlining data in fields i.e., Name, Phone, Birth Date and State will clean the data permanently which will assist in fast job processing, save analysts time for investigation etc. Secondly, you can see row#5 has been removed as, if the analysis is on Date then Year is missing from row#5 so this row will not assist in modelling.

Normally Data Cleansing and Data Enrichment are confused with each other.

‘Data Enrichment is about enhancing the data whereas Data Cleansing is about correcting the data.’

Data Cleansing has been covered in one of the previous topics, so we will focus only on Data Enrichment. As the definition explains itself, Data Enrichment is about adding more data into your first-hand data set from other internal or third-party data sources to add additional context. In this era of social media and unstructured sources, there is always additional data available which can enhance the context of your data generated from internal systems.

‘Data Enrichment must depend on Business Goals.’

For example, if a brand wants to open a new branch, it can buy geographical data from third parties. If a bank wants to offer Credit Cards deals, it can buy crediting rating data from third parties.

Please note, there can be many sources of data sets for Data Enrichment and of course not all can be Enriched in your data stores. Just like any other implementation, Data Enrichment must follow business requirements.