Knowledge Hub

All these terms like Data Integrity, Data Quality, Data Scrubbing, Data Cleansing and Data Cleaning are very interrelated.

• Cleaning: You eat something, and you get something on your face, you clean your face. You are painting something, and you get splashes of paint on your face, you clean your face. You are walking and you feel the dust, you wash and clean your face.
  SAME WAY, data cleaning can be done at a very high level by Business Users themselves as it requires little effort e.g., export data in excel, clean it, and send it downstream.
• Cleansing: You went on safari or played a football match on muddy ground, now you must use some face cream to cleanse your face plus you have to spend 10-15 minutes. Then it is better to ask someone or ideally go to the parlor for good face cleansing.
  SAME WAY, data cleansing can be done by business users themselves but ideally IT SME(s) can write SQL queries at the database level to cleanse data so next time when you extract, you get the right cleansed data.
• Scrubbing: This is the best level to achieve the highest Data Quality. Coming back to the same example, you need to attend a wedding or birthday party or any kind of valuable event, now you will go to a proper parlor, you will find the best worker, you will select the best brand of creams (based on your pocket vs the event you are attending) to scrub your face skin. You will make sure your face has been properly cleaned, cleansed, and scrubbed, and you will make sure you look ready for the event.
  SAME WAY, data scrubbing is to make sure your data is near to perfect, you will find tools that can do this for you, you will find good resources for this task, you will make sure the findings are implemented from Source to Downstream systems.

We touched base with Data Mining and Data Science, it will be ideal to discuss Data Dredging, Snooping, p-hacking, and Fishing. New terms right 😊?

Data Dredging, Snooping and Fishing all refer to the same behavior of data analysis BUT without proper hypothesis and relationship among datasets.

Data mining finds results based on the correlation of data in large data sets, but Data Dredging, Snooping, p-hacking, and Fishing find results based on chance methodology. In other words, Data Dredging, Snooping, p-hacking, and Fishing share the results which require more investigation. These sometimes bypass Data Mining techniques and come up with immature conclusions. On occasions, it shows more details about something than it contains.

There are two types of Fact Findings in any analysis that ultimately assists in DSS i.e., 1) known facts 2) unknown facts. Data mining techniques extract Known Facts as it’s done only on structured datasets and mostly, we know what's there in structured data like relationships between tables and relationships of data within the same table. But that's Not True, sometimes, even in structured data there are Unknown Facts and that’s where Data Dredging, Snooping and Fishing techniques come into the action.

Data Wrangling or Data Mangling or Data Munging are used interchangeably and do the same function.

Data Wrangling is something that every ETL developer, Data Engineer, Data Analyst etc. do daily. It is something that is done before Data Mining, Data Science, Business Intelligence or Business Analytics.

In simple words, Data Wrangling is to prepare data for processing. Note, prepare data for processing, not process data. Yes, you heard it correctly, sometimes Data Wrangling techniques are used for permanent data fixes and sometimes to prepare data for specific use cases.

Let's take an example, refer to the tables above, fixing and streamlining data in fields i.e., Name, Phone, Birth Date and State will clean the data permanently which will assist in fast job processing, save analysts time for investigation etc. Secondly, you can see row#5 has been removed as, if the analysis is on Date then Year is missing from row#5 so this row will not assist in modelling.

Normally Data Cleansing and Data Enrichment are confused with each other.

‘Data Enrichment is about enhancing the data whereas Data Cleansing is about correcting the data.’

Data Cleansing has been covered in one of the previous topics, so we will focus only on Data Enrichment. As the definition explains itself, Data Enrichment is about adding more data into your first-hand data set from other internal or third-party data sources to add additional context. In this era of social media and unstructured sources, there is always additional data available which can enhance the context of your data generated from internal systems.

‘Data Enrichment must depend on Business Goals.’

For example, if a brand wants to open a new branch, it can buy geographical data from third parties. If a bank wants to offer Credit Cards deals, it can buy crediting rating data from third parties.

Please note, there can be many sources of data sets for Data Enrichment and of course not all can be Enriched in your data stores. Just like any other implementation, Data Enrichment must follow business requirements.

We all have heard about the Data Driven Organization concept, so here is how it’s done. Data Democratization will make it happen.

‘Data Democratization is the foundation for Data Self-Service Environment.’

Data Democratization/ Data Self-Service is not a tool, nor it’s a technology. It’s about the working behavior of everyone within an organization with respect to data. Traditionally, whenever we need data, we go to IT and ask for some reports or dashboards. We normally tell them, this is our KPI, in other words, we tell them in simple English e.g., we need sales reports about a product from a particular region. Now, it’s IT’s job to find out where relevant data is residing, it’s their job to extract, process and prepare data for business users in the shape of a report or a dashboard. The worrying part is e.g., after IT takes 3 months to prepare a dashboard for Business User and when user says, please include 1 more column 😊, now the whole exercise starts from scratch. Data Democratization fills this gap.

Data Liberalization and Data Democratization are very confusing terms as both work together and most of the time mixed with each other.

‘Data Liberalization ideally is a prerequisite for Data Democratization.’

The aim for Data Democratization is to remove the fear factor about data. Allow everyone to ask analytical questions. To have data access availability for everyone. Allow everyone to raise questions about data and then suggest resolutions, if possible.

Data Liberalization is about providing data in such a way that it should be reusable by users. It can be in any formal i.e., via API, via BI tools, via DaaS etc.

When we say, ‘Aim is to have data access availability for everyone.’ doesn’t mean show everything, of course with access control. Data Liberalization completes the cycle to make sure whatever user is accessing, is USABLE for them rather a pile of junk data.

Data Driven Organization concept can’t be completed just by Data Democratization, Data Liberalization is also a very crucial component to achieve.

Data Literacy is the key for all Data Initiatives.

We know the difference between Business Intelligence (BI) developers and Business Analysts (BA). In previous eras, BA used to talk to business, create KPI(s) and pass it to BI developers to create reports and dashboards. 90% of BI developers would not know what a report or a dashboard is for. They won’t know the impact of good and bad reports. They won’t know the ROI of information this report, or dashboard can produce due to which they don’t care whether BA has provided right KPI(s) or not. This era was not a Data Era, in that era Reports or Dashboards were good to have. Most of the decision making used to happen in reports extracted in excels 😊.

Now time has changed, we have entered the Analytics era as compared to previous era of Analysis. Difference between Analysis and Analytics is simple but tricky i.e., Analysis is about finding Whys’ and What’s whereas Analytics is about finding How’s from a dataset. In other words, in datasets, Analysis search Questions and Analytics propose Answers.

Data Driven Organization is a new Buzz word in market where every organization wants to tag this Key Word with their organization.

Traditionally, we use data to make decisions. Decisions at top level for organizational revenue generation planning or at division level like whether division of a particular product is performing well or is it achieving its planned ROI, is it generating profits for the company.

Above mentioned attributes or properties doesn’t make you Data Driven Organization, above are previous era organizational behavior. Now, the current era is all about Data, everything within an organization must follow data decisions. When we data decisions we are talking about Daily Chores. We are talking about how employees hiring, and firing is done, we are talking about how transfers are done, we are talking about how printer Ink or Papers are orders, we are talking about procurement works, we are talking about how hardware, software or infrastructure resources are provisioned or de-provision.

Data Entitlement is normally mixed with Data Authorization which is not 100% accurate. There is no doubt, there are few places where both can be used interchangeably but mostly, they should not.

Entitlement is something which you inherit, or which comes as a default or which you get along-with something else whereas Authorization comes because of an action, or it expects an action in future. For example, you are ENTITLED to something and at the same time you are AUTHORIZED to do X, Y and Z things.

Before we understand the difference between Authorization and Entitlement, it’s very importance to understand what Prescriptive and Descriptive behaviors are.

• Prescriptive means, how it should be, and an action is expected.
• Descriptive means, how it is.

‘Authorization has a Prescriptive behavior whereas Entitlement has Descriptive behavior.’

Let’s try to understand with an example, at work you must have heard, you are ENTITLED to 20 annual leaves and maybe 12 medical leaves etc. But when you apply for a leave, your manager has the AUTHORITY to accept or reject your leave.

Authentication is the basic security feature of any organization. An organization without Authentication is like gold mine without any guard.

Authentication and Authorization works hands-in-hand and are a MUST-TO-HAVE in any organization or in any system or in any application.

Authentication is majorly tagged with Security whereas Authorization is majorly tagged with Privacy.

Let’s try to understand with an example, for example, in a hospital close relative, family or friends are allowed to come and meet the patient, but all these people normally are AUTHENTICATED or stopped by the guard at the main gate who checks their ID cards or anything which can assure that they are legitimate people e.g., guard can ask, why they are here etc. Once they are in, at the main hospital counter someone again asks them who they want to meet, if the patient’s name is there in system, then they are allowed to go ahead.