Knowledge Hub

As the name says itself, Data Splitting is about dividing data into multiple datasets which can be for multiple departments, for multiple regions, for multiple time periods etc. but here we will discuss how it’s used for data science?

Data Splitting for data science activities is an important aspect of creating, processing, validating, and testing machine learning models.

In Data Science, mostly 2 approaches are adopted to split datasets.

• Approach#1: Dataset are split in 2 subsets i.e., 1) Training 2) Testing
• Approach#2: Dataset are split in 3 subsets i.e., 1) Training 2) Validating 3) Testing

Let’s understand with an example, consider a dataset contains 100,000 bank customers, now a created model’s aim is to tag each customer under three categories e.g., Silver, Gold or Premium based on their credit history, buying behavior, transaction values etc.

Now let’s see how both approaches on with above example.

This Feature is not that traditional feature 😊. Here, we will be discussing Feature which is an integral part of Data Science activity. I am sure non-data scientists or non-data citizens might not be aware of this term.

We know what Data Science is, it is about running algorithms and machine learning models (ML models) on structured, semi and unstructured data sets for analytics.

For any ML model there is a basic requirement of data, right? And we all know Data Scientists love to play with Raw Data. Now, Raw Data means irrelevant data as well, so Data Scientists need Raw BUT relevant Raw Data. In Big Data, we know data sizes are huge and to train models, one doesn’t need all Raw Data. So, what data scientists do, they themselves or ideally data engineers clean and filter the data, in other words do Data Wrangling, before feeding into ML models.

For example, there is a table with 1 million rows containing data around the globe with all kinds of customers, but ML models is supposed to predict customers buying behavior for a particular country and for a particular set of products. Means, we don’t need to feed 1 million rows to our ML model so now we will create another table (FEATURE) where we will only keep data for that country and for required set of products and now make that table as source for our ML models.

Data Anomalies are about any kind of unusual behavior which can degrade the quality of data. Please note, data quality is not only about dirty data but also about not being able to insert, update and delete required data.

Type of Data Anomalies

• Insertion: If one is unable to enter quality data in database.
• Update: If one is unable to update relevant data or able to update irrelevant data.
• Delete: If one is unable to delete relevant data or able to delete irrelevant data.
• Redundancy: If there are duplicate records in a table.

Data Normalization and database constraints play a vibrant role to make sure data anomalies doesn’t occur. To start with, let’s say there is a table with employee ID, name, department, and salary details. First wrong approach we can noticed i.e., this table is in de-normalized form, by right there should be separate tables for department and salary data using normalization process. Normalization and Denormalization are covered, in detail, in data modelling topic.

By now, we all know what Semantic Layer is, IT defines data elements for business ease. Then there came the era for Business folks defines data elements themselves based on their own understanding and now the latest approach is going towards Machine Learning Models define data element based on open-ended requirements.

A major difference between first two approaches vs Augmented Analytics is, first two were derived by humans either by IT or Business folks so there is always a factor of human era and at the same time human can be biased base on knowledge and information they possess about the business domain. On the other hand, Augmented Analytics is based on automated Machine Learning which learns the data elements while new data sets keep coming in.

The first approach was static as requirements were always coming from business and implemented by IT folks, so any change had a round robin effect.

There are many ways to achieve data privacy like data generalization, data swapping, data masking, data encryption etc., all these techniques change original data which of course is not desired by most. All these are done to ensure data should not fall into wrong hands.

Data Synthetic is something which will really solve the problem statement like anonymize data before sharing with developers, data science teams for data models etc.

Data Synthetic is about generating new dataset using computer simulations. This dataset contains fake data with the right properties or attributes. For example, customer information is required in almost every application or system, but it can’t be shared to protect individual identity. Data Synthetic tools create a dummy dataset for customers with name, email, contact number, family details etc. but nothing related to original data. This way, developers can build the data pipelines and data scientists can build their data models without thinking of breaching any regulatory or compliance clauses.

Data Vectorization is something very common nowadays, especially since the inception of Big Data or Hadoop. It’s not like it was not in-use in the past, but I would say it was not famous.

All of us must have heard about MPP (Massive Parallel Processing) right? But do we know how it works at the back end? Data Vectorization is about enabling parallel processing to fetch data.

To understand MPP, let’s refer to four types of instructions to pull data in computer architecture i.e., SISD, SIMD, MISD and MIMD (all explained in separate topic). In this topic, we will be referring to SISD and SIMD only.

Traditionally, a computer or machine or a server works in SISD mode i.e., Single Instruction Stream and Single Data Stream means each instruction fetches required data one by one.

In Data Vectorization, we change our approach to MPP mode i.e., computer or machine or server starts working in SIMD mode i.e., Single Instructure and Multiple Data, means if one query is executed and data is residing in multiple data nodes, data from all nodes will be pulled in parallel making computation must faster as compared to SISD.

Data Drift concept is used in Data Science practice, and it is one of the most expected behaviors in every data modelling. One must have automated ways to keep tracking and identifying Data Drifts as after Data Drift, there are high chances the model will not produce or predict the expected accuracy which was planned. Wrong prediction means wrong decisions.

We all know what a Drift is. Drift is about shifting or moving from one point to another but not in a standard approach. This term is mostly referred in Data Science practice, to evaluate trained vs actual., performances of Machine Learning (ML) Models.

Data Drift is also known as covariate shift, virtual drift, or virtual concept drift. When we create and run a ML model, just like any traditional query, it inputs a dataset and outputs a dataset. In ` ML modelling, we select certain data elements to build its logic. Data Drift is about change of data within same select data element.

Data Imputation is majorly used in Data Science domain.

‘Imputation means, fill missing data.’

Data Imputation normally comes into play so business rules can be imposed on all data rather on subsets of data. Before Machine Learning Model runs, data imputation is done by either missing data is filled up or the whole row is removed.

Missing value can create many kinds of issues e.g., during categorizing, classifying, clustering etc., if data is missing then results will not be 100%.

Generally, in numeric datasets, missing values are replaced with MEAN values. Having said that, there many ways to deal with missing data.

Few Imputations Techniques

• The first technique is where data is missing at random basis but before using it, it is completely removed from the dataset.
• The second technique is also where data is missing at random basis but here missing values are replaced referring to the other data points available with current dataset.
• Third technique is where data is not missing at random basis and mainly replaced using Mean/Median/Mode values.

Data Fusion, another term overlapping with many other already in-place terms.

‘Data Fusion is about pulling same nature of data from multiple source systems to co-relate with each other to produce one unified output.’

For example, data for a particular product can come from POS (point of sales), from shops, from online stores etc. At the same time, people can talk about that product on social media, in seminars, webinars etc. Fusion of all those data elements can produce valuable information about that product for sales and marketing teams.

Yes, looking into image you can make out same thing can be achieved in Data Warehouse, Data Mart, Data Lake, Data Lakehouse, Data Vault/ Business Vault etc., and you are not wrong. All above-mentioned Data Store can be used for Data Fusion.

The concept of text and embedded vectorization in machine learning can be a bit confusing when compared to general data or text vectorization. Both approaches share the overarching goal of accelerating tasks, but they operate in different ways.

General Vectorization: The primary aim is to convert a single action on a single data element (SISD) into a single action on multiple data elements. This involves transforming the data in a way that allows for parallel processing, improving efficiency in handling multiple data elements simultaneously.

Text Vectorization in Machine Learning: In the context of machine learning, text vectorization specifically deals with converting textual or character data into numerical representations. Once the text or characters are converted into numbers, the embedding process comes into play. This process involves using machine learning and deep learning techniques to learn from these numerical representations, facilitating faster and more effective results.