Knowledge Hub

In the current era, people are confused between AI vs Data Science.

‘This is a very controversial topic.’

If you are clear about Data Science, then understanding AI will not be a problem. In the current era, with huge datasets either its structure, semi, or unstructured data, we create Data Science Models to analyze and predict. So, when it’s done by Humans, it’s called Data Science but when models are matured and are passed on to the systems or machines to run itself, train itself, fix itself, analyze itself and predict itself then it becomes Artificial Intelligence (AI).

AI is much more than that but here aim is not to explain or elaborate in detail like what AI is, here aim is to explain the difference between AI and Data Science at a very high level.

To understand Data Algorithms, let's quickly know what an algorithm is. It is a procedure consisting of multiple steps with a set of instructions for a desired result. It is independent of any language, in other words, an algorithm can be implemented in any language.

Algorithm Characteristics

• Unambiguous: It should be clear and unambiguous. Each of its steps, and its inputs/outputs should be clear and must lead to only one meaning.
• Input: It should have 0 or more well-defined inputs.
• Output: It should have 1 or more well-defined outputs and should match the desired output.
• Finiteness: It must terminate after a finite number of steps.
• Feasibility: It should be feasible with the available resources.
• Independent: It should have step-by-step directions, which should be independent of any programming code.

Now from a data point-of-view, how the algorithm behaves.

• Search: Algorithm to search an item in a data structure.
• Sort: Algorithm to sort items in a certain order.
• Insert: Algorithm to insert an item in a data structure.
• Update: Algorithm to update an existing item in a data structure.
• Delete: Algorithm to delete an existing item from a data structure.

As the name says itself, Data Splitting is about dividing data into multiple datasets which can be for multiple departments, for multiple regions, for multiple time periods etc. but here we will discuss how it’s used for data science?

Data Splitting for data science activities is an important aspect of creating, processing, validating, and testing machine learning models.

In Data Science, mostly 2 approaches are adopted to split datasets.

• Approach#1: Dataset are split in 2 subsets i.e., 1) Training 2) Testing
• Approach#2: Dataset are split in 3 subsets i.e., 1) Training 2) Validating 3) Testing

Let’s understand with an example, consider a dataset contains 100,000 bank customers, now a created model’s aim is to tag each customer under three categories e.g., Silver, Gold or Premium based on their credit history, buying behavior, transaction values etc.

Now let’s see how both approaches on with above example.

This Feature is not that traditional feature 😊. Here, we will be discussing Feature which is an integral part of Data Science activity. I am sure non-data scientists or non-data citizens might not be aware of this term.

We know what Data Science is, it is about running algorithms and machine learning models (ML models) on structured, semi and unstructured data sets for analytics.

For any ML model there is a basic requirement of data, right? And we all know Data Scientists love to play with Raw Data. Now, Raw Data means irrelevant data as well, so Data Scientists need Raw BUT relevant Raw Data. In Big Data, we know data sizes are huge and to train models, one doesn’t need all Raw Data. So, what data scientists do, they themselves or ideally data engineers clean and filter the data, in other words do Data Wrangling, before feeding into ML models.

For example, there is a table with 1 million rows containing data around the globe with all kinds of customers, but ML models is supposed to predict customers buying behavior for a particular country and for a particular set of products. Means, we don’t need to feed 1 million rows to our ML model so now we will create another table (FEATURE) where we will only keep data for that country and for required set of products and now make that table as source for our ML models.

All these terms like Data Integrity, Data Quality, Data Scrubbing, Data Cleansing and Data Cleaning are very interrelated.

• Cleaning: You eat something, and you get something on your face, you clean your face. You are painting something, and you get splashes of paint on your face, you clean your face. You are walking and you feel the dust, you wash and clean your face.
  SAME WAY, data cleaning can be done at a very high level by Business Users themselves as it requires little effort e.g., export data in excel, clean it, and send it downstream.
• Cleansing: You went on safari or played a football match on muddy ground, now you must use some face cream to cleanse your face plus you have to spend 10-15 minutes. Then it is better to ask someone or ideally go to the parlor for good face cleansing.
  SAME WAY, data cleansing can be done by business users themselves but ideally IT SME(s) can write SQL queries at the database level to cleanse data so next time when you extract, you get the right cleansed data.
• Scrubbing: This is the best level to achieve the highest Data Quality. Coming back to the same example, you need to attend a wedding or birthday party or any kind of valuable event, now you will go to a proper parlor, you will find the best worker, you will select the best brand of creams (based on your pocket vs the event you are attending) to scrub your face skin. You will make sure your face has been properly cleaned, cleansed, and scrubbed, and you will make sure you look ready for the event.
  SAME WAY, data scrubbing is to make sure your data is near to perfect, you will find tools that can do this for you, you will find good resources for this task, you will make sure the findings are implemented from Source to Downstream systems.

We touched base with Data Mining and Data Science, it will be ideal to discuss Data Dredging, Snooping, p-hacking, and Fishing. New terms right 😊?

Data Dredging, Snooping and Fishing all refer to the same behavior of data analysis BUT without proper hypothesis and relationship among datasets.

Data mining finds results based on the correlation of data in large data sets, but Data Dredging, Snooping, p-hacking, and Fishing find results based on chance methodology. In other words, Data Dredging, Snooping, p-hacking, and Fishing share the results which require more investigation. These sometimes bypass Data Mining techniques and come up with immature conclusions. On occasions, it shows more details about something than it contains.

There are two types of Fact Findings in any analysis that ultimately assists in DSS i.e., 1) known facts 2) unknown facts. Data mining techniques extract Known Facts as it’s done only on structured datasets and mostly, we know what's there in structured data like relationships between tables and relationships of data within the same table. But that's Not True, sometimes, even in structured data there are Unknown Facts and that’s where Data Dredging, Snooping and Fishing techniques come into the action.

Data Wrangling or Data Mangling or Data Munging are used interchangeably and do the same function.

Data Wrangling is something that every ETL developer, Data Engineer, Data Analyst etc. do daily. It is something that is done before Data Mining, Data Science, Business Intelligence or Business Analytics.

In simple words, Data Wrangling is to prepare data for processing. Note, prepare data for processing, not process data. Yes, you heard it correctly, sometimes Data Wrangling techniques are used for permanent data fixes and sometimes to prepare data for specific use cases.

Let's take an example, refer to the tables above, fixing and streamlining data in fields i.e., Name, Phone, Birth Date and State will clean the data permanently which will assist in fast job processing, save analysts time for investigation etc. Secondly, you can see row#5 has been removed as, if the analysis is on Date then Year is missing from row#5 so this row will not assist in modelling.

Normally Data Cleansing and Data Enrichment are confused with each other.

‘Data Enrichment is about enhancing the data whereas Data Cleansing is about correcting the data.’

Data Cleansing has been covered in one of the previous topics, so we will focus only on Data Enrichment. As the definition explains itself, Data Enrichment is about adding more data into your first-hand data set from other internal or third-party data sources to add additional context. In this era of social media and unstructured sources, there is always additional data available which can enhance the context of your data generated from internal systems.

‘Data Enrichment must depend on Business Goals.’

For example, if a brand wants to open a new branch, it can buy geographical data from third parties. If a bank wants to offer Credit Cards deals, it can buy crediting rating data from third parties.

Please note, there can be many sources of data sets for Data Enrichment and of course not all can be Enriched in your data stores. Just like any other implementation, Data Enrichment must follow business requirements.

We all have heard about the Data Driven Organization concept, so here is how it’s done. Data Democratization will make it happen.

‘Data Democratization is the foundation for Data Self-Service Environment.’

Data Democratization/ Data Self-Service is not a tool, nor it’s a technology. It’s about the working behavior of everyone within an organization with respect to data. Traditionally, whenever we need data, we go to IT and ask for some reports or dashboards. We normally tell them, this is our KPI, in other words, we tell them in simple English e.g., we need sales reports about a product from a particular region. Now, it’s IT’s job to find out where relevant data is residing, it’s their job to extract, process and prepare data for business users in the shape of a report or a dashboard. The worrying part is e.g., after IT takes 3 months to prepare a dashboard for Business User and when user says, please include 1 more column 😊, now the whole exercise starts from scratch. Data Democratization fills this gap.

Data Liberalization and Data Democratization are very confusing terms as both work together and most of the time mixed with each other.

‘Data Liberalization ideally is a prerequisite for Data Democratization.’

The aim for Data Democratization is to remove the fear factor about data. Allow everyone to ask analytical questions. To have data access availability for everyone. Allow everyone to raise questions about data and then suggest resolutions, if possible.

Data Liberalization is about providing data in such a way that it should be reusable by users. It can be in any formal i.e., via API, via BI tools, via DaaS etc.

When we say, ‘Aim is to have data access availability for everyone.’ doesn’t mean show everything, of course with access control. Data Liberalization completes the cycle to make sure whatever user is accessing, is USABLE for them rather a pile of junk data.

Data Driven Organization concept can’t be completed just by Data Democratization, Data Liberalization is also a very crucial component to achieve.