Knowledge Hub

Data discovery involves the collection and evaluation of data from various sources and is often used to understand trends and patterns in the data. But understanding trends and patterns cannot be simply done based on data coming from sources such as.

• Data can have Null values.
• Data can have different types of date formats.
• Data can have spelling mistakes.
• Data can have special characters.
• Data can have different codes for the same product e.g., a Glass code can be GLS in one source system and GS in another. So, it's very important to streamline the data sets before presenting them to end users.
• Data can have duplicates.
• And many more....

Data Discovery is different for different roles i.e., for Data Scientist or Data Analyst or Business End User or for C-Level.

• Data Scientists mostly prefer raw data without any data cleaning, cleansing, and scrubbing for Data Dredging, Snooping, p-hacking, and Fishing [there are separate topics on these terms].
• Data analysts mostly prefer data after it’s at least cleaned and cleansed.
• Business End Users and C-Level expect data to be cleaned, cleansed, scrubbed, wrangled, and munged [there are separate topics on these terms] before it reaches them for decision making. Well, a fair expectation.

Heat Map is a graphical representation of objects at any specific location. Nowadays, it is used to identify highly populated area(s) during Covid-19 era. Heat maps make it easy to visualize complex data and understand it briefly.

Now, the question is how Heat Map software knows where the location is. When we open any webpage or any app like Facebook or Twitter etc., via mobile or laptop, our location is picked up. That location is based on Lat and Long (Latitude and Longitude) values. These values are then sent to databases which are then picked up by the Heat Map software for graphical presentation.

• Latitude is the measurement of distance north or south of the Equator. It is measured with 180 imaginary lines that form circles around the Earth east-west, parallel to the Equator.
• Longitude is the measurement east or west of the prime meridian. Longitude is measured by imaginary lines that run around the Earth vertically (up and down) and meet at the North and South Poles. These lines are known as meridians. Each meridian measures one arc degree of longitude. The distance around the Earth measures 360 degrees.

DIKW model

‘Data --> Information --> Knowledge --> Wisdom’

Generally, Data, Information, Knowledge, and Wisdom are arranged in the above sequence. How about considering another angle and trying to arrange these 4 terms in different sequence?

For me Information and Knowledge both comes from Data and Wisdom comes from both Information and Knowledge.

Difference between Data and Information?

As we all know, now a days, Data is everywhere either it is generating from System/ Mobile Applications, Online Websites/ Portals, Videos, Audios, logs etc.

• Is all Data equal to Information? No, it’s NOT.
• Can all Data become Information? Yes, it CAN, based on the nature of the business or requirement.

To answer above, one must first understand what Information is. Information is tightly coupled with what one wants to know. You must have heard this question on many occasions or events or counters etc. like What Information do you need?

Monetization is about generating benefit/ revenue/ profit from anything.

‘Data Monetization is about generating revenue from Data.’

In this era, social media is a very dominant channel to interact with customers. Platforms like WhatsApp, Facebook, Twitter, TikTok, Instagram etc. all provide free services to their customers. Have you ever noticed, if you are not paying a single penny to them then how they are generating money and how they are running their business?

The response to the above is by Data Monetization. They sell the data that’s generated by their customers. Having said that, please note, there are regulations that they can’t share personal details of their customers with anyone. But that doesn’t mean they can’t sell how many people are viewing what, how many are buying, how many are spending how much on which product category, from which location they are logged in etc.

For example, on Amazon, customers can buy anything they like. Now at the back end, Amazon must be recording what you are buying and what you are selling. Based on this Information, they can very well analyze which product is more viewed on their portal or what is the buying trend of a particular set of products in a particular season. Like, in December, they can see the trend of products related to Christmas are in great demand. They can sell information like WHICH & HOW MANY PRODUCTS WILL BE IN DEMAND IN COMING DECEMBER based on last few years trend to the product owner to plan their production. This information is GOLD for product owners, they can plan their raw material, work force, timeline and many more activities.

Point-to-Point model (PP) connects all applications with each other e.g., if there are 5 applications then there will be many connections as depicted in below picture.

The Hub-and-Spoke model (SP) creates a new centralized connection points called as Hub and all applications will ONLY connect with centralized ONE Hub.

We can clearly see Hub-and-Spoke model is much better in access management, flexible, cheaper, secure etc. as compared to Point-to-Point model.

Examples of Hub-and-Spoke model are Data Warehouse (dimensional), Data Hub, Data Fabric etc. which serves as single point of truth for all enterprise users.

Key Items

• User Access: SP model can management user access matrix at centralized place as compared to PP model.
• Security: As all users will be connecting to one centralized Hub, risk of security breach will be less in SP model as compared to PP model
• Flexibility: In SP model, Hub is easy to incorporate those new requirements which doesn’t need to implement in source systems.
• Cheaper: In SP model, though Hub implementation requires initial investment but once it’s implemented all new requirements cost will have much cheaper impact as all changes will go in Hub as compared to each application in PP model.

This is a principle rather a process to manipulate data, but it holds a very significant value in terms of managing and securing data.

Most of us mix Critical Data Elements with elements or data points identified during Data Quality exercise. Though those are also critical but when we talk about ‘Critical Data Element’ in Data Strategy, we categorize those elements which can have negative impact on the organization.

Critical Data Element can be any regulatory requirements, any legal or compliance requirements, those data elements which can negatively impact the image of company in general public, those data elements which if are found inaccurate make stakeholders’ loss their confidence on data.

Personal Information of customers is one of the most critical data elements in most of the organization. Credit or Debit Card details in the Banking sector are critical. Location details in Telecommunication are critical etc.

This is also a very confusing term in Data domain, a lot of people are not sure how to assure that everyone follows Ethics in their day-to-day activities while interacting with Data. Just like CDE (Critical Data Element), Data Ethics is also a principle and must be managed at organization level and followed at individual level.

Now, the big question is which data falls under the ambarella where Data Ethics must be followed. There are multiple factors involved and everyone has his/ her own level of ethics to handle based on their roles and responsibilities.

Data Ethics is also a term which if not taken care of properly, can damage a company’s image with the general public. To make sure organizations follow ethics, different regulatory bodies in different parts of the world and in different sectors give guiding principles to follow. It doesn’t stop here, if the organization doesn’t end up following all the measures asked-for by those regulatory bodies then they can be penalized with heavy amounts as penalties.

Data Anomalies are about any kind of unusual behavior which can degrade the quality of data. Please note, data quality is not only about dirty data but also about not being able to insert, update and delete required data.

Type of Data Anomalies

• Insertion: If one is unable to enter quality data in database.
• Update: If one is unable to update relevant data or able to update irrelevant data.
• Delete: If one is unable to delete relevant data or able to delete irrelevant data.
• Redundancy: If there are duplicate records in a table.

Data Normalization and database constraints play a vibrant role to make sure data anomalies doesn’t occur. To start with, let’s say there is a table with employee ID, name, department, and salary details. First wrong approach we can noticed i.e., this table is in de-normalized form, by right there should be separate tables for department and salary data using normalization process. Normalization and Denormalization are covered, in detail, in data modelling topic.

Data Surfing is always confused with Data Analysis. There is a very thin line between Data Analysts and Data Surfers.

Data Surfing is normally done by non-technical people like management who doesn’t know how to write any kind of code to access data at database level.

Data Surfers normally stays at the level of Dashboards and Reports where they just click here and there to explore data for required information.

Business Intelligence (BI) tools came into being to cater Data Surfers World where data was required to be presented for non-technical people who could view data for decision making.

Era(s) for Data Surfers

• Static Reports: In the beginning, business users used to share their data requirements with IT who then used to create dashboards and reports. Those dashboards/ reports and data was not changeable, it was in paper format or in excel format or in text format etc., means after Go-Live if business want one more column or if latest data was required then the whole development cycle had to restart from scratch.
• Dynamic Reports: In the second era, BI tools came-in to cater for this gap. To start with, dashboards/ reports were still not changeable, but data was directly connected with backend databases, so data was always latest at front-end. This was a huge jump, as now, to view new data developer don’t have to recreate front-end again and again.
• Adhoc Reports: Latest era is where dashboards/ reports are also changeable. Now, end users and management can drag and drop tables and columns in dashboards/ reports.

“Semantic” is an ancient Greek word: σημαντικός sēmantikós, "significant" is the study of reference, meaning, or truth.

In Data World, Semantic layer is built for the sole purpose of supporting business users.

Semantic layer is to convert complex data models into simple and user-friendly interface for business users.

A semantic layer is the representation layer on top of databases, schema(s), tables and columns in such a way that Business users who have no technical know-how should be able to drag and drop required details on Dashboards. For drag and drop, first, one must assign business names to all databases, schema(s), tables and columns before exposing in BI tools. For example, there can be a column name as Date but the data in this column can be Date_Of_Birth or Transaction Date or Financial_Year_End_Data. Converting data models, column/ table/ schema/ database names as per business understanding is semantic process.