Knowledge Hub

As the name says itself, Data Splitting is about dividing data into multiple datasets which can be for multiple departments, for multiple regions, for multiple time periods etc. but here we will discuss how it’s used for data science?

Data Splitting for data science activities is an important aspect of creating, processing, validating, and testing machine learning models.

In Data Science, mostly 2 approaches are adopted to split datasets.

• Approach#1: Dataset are split in 2 subsets i.e., 1) Training 2) Testing
• Approach#2: Dataset are split in 3 subsets i.e., 1) Training 2) Validating 3) Testing

Let’s understand with an example, consider a dataset contains 100,000 bank customers, now a created model’s aim is to tag each customer under three categories e.g., Silver, Gold or Premium based on their credit history, buying behavior, transaction values etc.

Now let’s see how both approaches on with above example.

This Feature is not that traditional feature 😊. Here, we will be discussing Feature which is an integral part of Data Science activity. I am sure non-data scientists or non-data citizens might not be aware of this term.

We know what Data Science is, it is about running algorithms and machine learning models (ML models) on structured, semi and unstructured data sets for analytics.

For any ML model there is a basic requirement of data, right? And we all know Data Scientists love to play with Raw Data. Now, Raw Data means irrelevant data as well, so Data Scientists need Raw BUT relevant Raw Data. In Big Data, we know data sizes are huge and to train models, one doesn’t need all Raw Data. So, what data scientists do, they themselves or ideally data engineers clean and filter the data, in other words do Data Wrangling, before feeding into ML models.

For example, there is a table with 1 million rows containing data around the globe with all kinds of customers, but ML models is supposed to predict customers buying behavior for a particular country and for a particular set of products. Means, we don’t need to feed 1 million rows to our ML model so now we will create another table (FEATURE) where we will only keep data for that country and for required set of products and now make that table as source for our ML models.

All these terms like Data Integrity, Data Quality, Data Scrubbing, Data Cleansing and Data Cleaning are very interrelated.

• Cleaning: You eat something, and you get something on your face, you clean your face. You are painting something, and you get splashes of paint on your face, you clean your face. You are walking and you feel the dust, you wash and clean your face.
  SAME WAY, data cleaning can be done at a very high level by Business Users themselves as it requires little effort e.g., export data in excel, clean it, and send it downstream.
• Cleansing: You went on safari or played a football match on muddy ground, now you must use some face cream to cleanse your face plus you have to spend 10-15 minutes. Then it is better to ask someone or ideally go to the parlor for good face cleansing.
  SAME WAY, data cleansing can be done by business users themselves but ideally IT SME(s) can write SQL queries at the database level to cleanse data so next time when you extract, you get the right cleansed data.
• Scrubbing: This is the best level to achieve the highest Data Quality. Coming back to the same example, you need to attend a wedding or birthday party or any kind of valuable event, now you will go to a proper parlor, you will find the best worker, you will select the best brand of creams (based on your pocket vs the event you are attending) to scrub your face skin. You will make sure your face has been properly cleaned, cleansed, and scrubbed, and you will make sure you look ready for the event.
  SAME WAY, data scrubbing is to make sure your data is near to perfect, you will find tools that can do this for you, you will find good resources for this task, you will make sure the findings are implemented from Source to Downstream systems.

We touched base with Data Mining and Data Science, it will be ideal to discuss Data Dredging, Snooping, p-hacking, and Fishing. New terms right 😊?

Data Dredging, Snooping and Fishing all refer to the same behavior of data analysis BUT without proper hypothesis and relationship among datasets.

Data mining finds results based on the correlation of data in large data sets, but Data Dredging, Snooping, p-hacking, and Fishing find results based on chance methodology. In other words, Data Dredging, Snooping, p-hacking, and Fishing share the results which require more investigation. These sometimes bypass Data Mining techniques and come up with immature conclusions. On occasions, it shows more details about something than it contains.

There are two types of Fact Findings in any analysis that ultimately assists in DSS i.e., 1) known facts 2) unknown facts. Data mining techniques extract Known Facts as it’s done only on structured datasets and mostly, we know what's there in structured data like relationships between tables and relationships of data within the same table. But that's Not True, sometimes, even in structured data there are Unknown Facts and that’s where Data Dredging, Snooping and Fishing techniques come into the action.

Data Wrangling or Data Mangling or Data Munging are used interchangeably and do the same function.

Data Wrangling is something that every ETL developer, Data Engineer, Data Analyst etc. do daily. It is something that is done before Data Mining, Data Science, Business Intelligence or Business Analytics.

In simple words, Data Wrangling is to prepare data for processing. Note, prepare data for processing, not process data. Yes, you heard it correctly, sometimes Data Wrangling techniques are used for permanent data fixes and sometimes to prepare data for specific use cases.

Let's take an example, refer to the tables above, fixing and streamlining data in fields i.e., Name, Phone, Birth Date and State will clean the data permanently which will assist in fast job processing, save analysts time for investigation etc. Secondly, you can see row#5 has been removed as, if the analysis is on Date then Year is missing from row#5 so this row will not assist in modelling.

Normally Data Cleansing and Data Enrichment are confused with each other.

‘Data Enrichment is about enhancing the data whereas Data Cleansing is about correcting the data.’

Data Cleansing has been covered in one of the previous topics, so we will focus only on Data Enrichment. As the definition explains itself, Data Enrichment is about adding more data into your first-hand data set from other internal or third-party data sources to add additional context. In this era of social media and unstructured sources, there is always additional data available which can enhance the context of your data generated from internal systems.

‘Data Enrichment must depend on Business Goals.’

For example, if a brand wants to open a new branch, it can buy geographical data from third parties. If a bank wants to offer Credit Cards deals, it can buy crediting rating data from third parties.

Please note, there can be many sources of data sets for Data Enrichment and of course not all can be Enriched in your data stores. Just like any other implementation, Data Enrichment must follow business requirements.

We all have heard about the Data Driven Organization concept, so here is how it’s done. Data Democratization will make it happen.

‘Data Democratization is the foundation for Data Self-Service Environment.’

Data Democratization/ Data Self-Service is not a tool, nor it’s a technology. It’s about the working behavior of everyone within an organization with respect to data. Traditionally, whenever we need data, we go to IT and ask for some reports or dashboards. We normally tell them, this is our KPI, in other words, we tell them in simple English e.g., we need sales reports about a product from a particular region. Now, it’s IT’s job to find out where relevant data is residing, it’s their job to extract, process and prepare data for business users in the shape of a report or a dashboard. The worrying part is e.g., after IT takes 3 months to prepare a dashboard for Business User and when user says, please include 1 more column 😊, now the whole exercise starts from scratch. Data Democratization fills this gap.

Data Liberalization and Data Democratization are very confusing terms as both work together and most of the time mixed with each other.

‘Data Liberalization ideally is a prerequisite for Data Democratization.’

The aim for Data Democratization is to remove the fear factor about data. Allow everyone to ask analytical questions. To have data access availability for everyone. Allow everyone to raise questions about data and then suggest resolutions, if possible.

Data Liberalization is about providing data in such a way that it should be reusable by users. It can be in any formal i.e., via API, via BI tools, via DaaS etc.

When we say, ‘Aim is to have data access availability for everyone.’ doesn’t mean show everything, of course with access control. Data Liberalization completes the cycle to make sure whatever user is accessing, is USABLE for them rather a pile of junk data.

Data Driven Organization concept can’t be completed just by Data Democratization, Data Liberalization is also a very crucial component to achieve.

Data Literacy is the key for all Data Initiatives.

We know the difference between Business Intelligence (BI) developers and Business Analysts (BA). In previous eras, BA used to talk to business, create KPI(s) and pass it to BI developers to create reports and dashboards. 90% of BI developers would not know what a report or a dashboard is for. They won’t know the impact of good and bad reports. They won’t know the ROI of information this report, or dashboard can produce due to which they don’t care whether BA has provided right KPI(s) or not. This era was not a Data Era, in that era Reports or Dashboards were good to have. Most of the decision making used to happen in reports extracted in excels 😊.

Now time has changed, we have entered the Analytics era as compared to previous era of Analysis. Difference between Analysis and Analytics is simple but tricky i.e., Analysis is about finding Whys’ and What’s whereas Analytics is about finding How’s from a dataset. In other words, in datasets, Analysis search Questions and Analytics propose Answers.

Data Driven Organization is a new Buzz word in market where every organization wants to tag this Key Word with their organization.

Traditionally, we use data to make decisions. Decisions at top level for organizational revenue generation planning or at division level like whether division of a particular product is performing well or is it achieving its planned ROI, is it generating profits for the company.

Above mentioned attributes or properties doesn’t make you Data Driven Organization, above are previous era organizational behavior. Now, the current era is all about Data, everything within an organization must follow data decisions. When we data decisions we are talking about Daily Chores. We are talking about how employees hiring, and firing is done, we are talking about how transfers are done, we are talking about how printer Ink or Papers are orders, we are talking about procurement works, we are talking about how hardware, software or infrastructure resources are provisioned or de-provision.