Knowledge Hub

‘Data Segmentation is about dividing dataset into multiple sections based on some requirements.’

Segments are defined based on many parameters e.g., in a business, segments are defined by accessibility, actionable, profitable etc.

For example, when we go into a mall, there is a segment/section for Food Court where multiple categories of food are available with different classification like expensive or cheap food.

Steps to improve Data Segmentation

• Enrich your data to increase data quality.
• Plan and replan your data segmentation criteria(s)
• Improve real time segmentation.
• Design segment based on customer prioritization and feedback.
• Integrate with as many data elements.

In business, what you can do by having Data Segmentation

• You can create deals based on customer buying behavior.
• You can improve your products arrangement.
• You can plan your marketing strategy.
• You can analyze customer feedback.

Data Labelling is about tagging something which is not identifiable coming from source. The Labelling can be on known data and on unknown data as well.

Let’s try to understand the relationship among all these 3 terms i.e., Data Labelling, Data Classification and Data Clustering.

• Data Clustering group the data based on existing parameter and falls under Unsupervised Learning.
• Data Labelling is the process of tagging data based on some defined parameter.
• Data Classification group the data based on manually LABELLED data and falls under Supervised Learning.

Since the era of unstructured dataset, data is coming in many formats. There are specialized tools which can tag or label the unstructured data set so it can become input for Machine Learning (ML) Models. There are also certain libraries used in ML Models to label unstructured data set which can then become input of another ML Model for Supervised Learning.

Data Annotation and Data Labelling works together to prepare dataset for Machine Learning Models in Data Science practice.

‘Data Annotation is the technique of doing Data Labelling.’

Data Annotation is a complete process with many steps to curate, whereas Data Labelling is ONE of the many steps in it.

Data Labelling is about tagging the data for training e.g., machine learning models whereas Data Annotation is the technique for tagging the data so ML models can reuse it as well. Data Labelling tags are used by ML models for future pattern recognitions whereas Data Annotation is about recognizing the data via a process.

Just like Data Labelling, Data Annotation is also for supervised machine learning to prepare data as an input and to store it for future reference. Not to mention, there are STILL many organizations doing manual Data Labelling. Data Annotation is about the process so Data Labelling can be automated for accuracy, eliminating human intervention. Bad Data Labelling can impact the outcome of applications.

Entropy term is normally used in Data Science domain.

• Where finding unexpected outcomes is the aim
• Where surprises are welcomed
• Where results are analyzed based on probability ratio, the lower the probability the better it is.
• Where informative information is something which wasn’t known

Types

• High Entropy means, more surprises, more unexpected values are considered more informative.
• Low Entropy means, less surprises, less unexpected values are considered less informative.

‘Entropy is also called as Extreme Disorder of values.’

Referring to the image, we can see at the starting point all the signs are MINUS, then in the middle there are 50/50 signs of PLUS & MINUS and right at the end all the signs are PLUS. At those extremes left, middle and right corners, the Entropy is at the lowest, so no surprises are expected so nothing much for Data Scientists to predict right. But in 2^nd, 4^th, and 5^th circles, it’s difficult to say how many PLUSES(s) and MINUS(s), if values are not visible, at this points Entropy is at the highest.

Epistemology, a term rooted in the Greek word episteme, meaning “knowledge,” and logos, meaning “discourse” or “communication,” explores the concept of “justified knowledge.” It addresses questions like “What do we know?”, “How can we claim to know something?”, “What validates our beliefs?”, and “How do we ascertain the certainty of our knowledge?”

In the realm of data, these questions take on a new dimension, especially in the Big Data era, where we store vast amounts of raw data, structured, semi-structured, and unstructured. Data science teams apply machine learning or deep learning models to extract insights, turning raw data into knowledge. However, to ensure that these insights align with their origins, a “justification of knowledge” is essential, ensuring that what we extract truly reflects what was expected. This process emphasizes evidence-based validation, where qualitative research relies on the quality of design and data collected.

Data Hierarchy is a known term and most of us are aware of it but sometimes, it’s confused with Data Taxonomy & Data Ontology.

Data Taxonomy and Data Ontology are metadata for the purpose of Data Governance whereas Data Hierarchies presents actual data. A very well-known and famous example is the Organizational Chart where hierarchy starts from CEO and goes down to Database Administrator in data domain.

• ‘Data Taxonomies apply to identified objects or phenomena.’’
• ‘Data Ontologies apply to terms, concepts, or categories.’
• ‘Data Hierarchy is a chain of relationship in actually data.’

Family tree is another simple example.

Data can be grouped in a way where data exploration and representation become easy. For example, at database level, hierarchy can be Database 🡪 Files/ Tables 🡪 Records 🡪 Fields/ Columns 🡪 Bytes 🡪 Bits.

Same line Data Taxonomy, Data Hierarchy also have single parent and multiple child relationship among the rows of dataset.

Data Anonymization, Data Pseudonymization or Data De-identification all worked the same way and related to Data Privacy.

Data Anonymization is about removing or modifying personal information before sharing with others so those people whom the data about remains anonymized.

‘Data Anonymization is about hiding the identity of a person to be undetectable.’

For example, while implementing a data solution it is required to share data with development team, so before sharing one must make sure data is anonymized, persons details are not exposed to development team members e.g., name, contact number, address, credit card number etc.

Data Anonymization is normally enforced by Regulatory Bodies and not following can result in heavy penalties and a bad reputation for any organization.

Techniques [All these terms are covered separately]

• Data Masking
• Data Pseudonymization
• Data Generalization
• Data Swapping
• Data Perturbation
• Data Synthetic
• Data Shuffling
• Data Scrambling
• Data Encoding
• Data De-identification

Data Identification is the pre-requisite for Data Anonymization. How?

Data Identification is opposite to Data Anonymization, in other words, first one must identify data before conducting Data Anonymization.

Data Identification is about those set of details which assist in identifying an entity or individual. These details can be direct details like name, contact number, address, credit card number etc. or can be indirect details like unique identification number which assist to connect with other datasets to identify an entity or individual.

Data Identification is a very crucial exercise in data migration project where the process is to first conduct data discovery then data identification and then data migration can take place.

Data Identification can be done in all kinds of datasets i.e., structure, semi or unstructured but recommended practice is to do it in structured datasets.

Other Benefits

Data Generalization/ Blurring is normally confused with Data Anonymization which is about removing or modifying personal information so the identity of a person can be undetectable. Data Generalization aim is same but using with different approach.

‘Data Generalization/ Blurring and Specialization is something like Zoom-Out vs Zoom-In or Precise vs Less-Precise.’

One of the main requirements for Data Generalization comes into play when you must maintain Data Sovereignty (explain in a separate topic) but keep the data originality. It is used to cater for data privacy.

‘Data Generalization summarized data in such a way that the usefulness of the data remains intact.’

Data Specialization is the right opposite of Data Generalization as it divides entity into multiple entities to further make sense of the data.

Example of Data Generalization: Referring to the image, Students and Patients both have Name and Mobile numbers, we can have summarized entity like Person which can host both Students and Patients details i.e., Roll_No and PID numbers. This way, there will be records of all the students and patients, but no one will be able to identify any student or patient.

Data Perturbation/ Swapping/ Shuffling/ Data Scrambling/ Data Obfuscation all use the same technique to hide the identity of an entity but keeping the essence of the data.

Data Swapping is about replacing the values of data at the same level of detail as compared to original data. In other words, cardinality (explained in a separate topic) of the replaced data is maintained.

• In Data Anonymization, identifiable data was removed.
• In Data Generalization, identifiable data is summarized.
• In Data Swapping, the level of identifiable data is replaced.

‘Data Swapping technique is majorly used by Data Scientists or Data Citizen who require data at the lowest level to create and train their Machine Learning Models.’

Let’s refer to the same Age example used in topic: Data Generalization/ Blurring and Specialization. For example, there can be an Age column with value: 26, 28, 31, 33, 37, 42, 42, 46, 48, 49, 54, 57, 57, 58, 59. In Data Swapping, to maintain the same level of detail i.e., ages can be swapped 26 with 59, 28 with 58, 32 with 57 and so on.