Knowledge Hub

Data Entitlement is normally mixed with Data Authorization which is not 100% accurate. There is no doubt, there are few places where both can be used interchangeably but mostly, they should not.

Entitlement is something which you inherit, or which comes as a default or which you get along-with something else whereas Authorization comes because of an action, or it expects an action in future. For example, you are ENTITLED to something and at the same time you are AUTHORIZED to do X, Y and Z things.

Before we understand the difference between Authorization and Entitlement, it’s very importance to understand what Prescriptive and Descriptive behaviors are.

• Prescriptive means, how it should be, and an action is expected.
• Descriptive means, how it is.

‘Authorization has a Prescriptive behavior whereas Entitlement has Descriptive behavior.’

Let’s try to understand with an example, at work you must have heard, you are ENTITLED to 20 annual leaves and maybe 12 medical leaves etc. But when you apply for a leave, your manager has the AUTHORITY to accept or reject your leave.

Self-Service and Data Enrichment are mostly confused with each other where data enrichment is just one part of one of the types of self-services.

Self-Service is about empowering business users and decision makers to conduct activities by themselves to shorten the end-to-end delivery time. At the same time, it’s too risky to allow people to go and make changes if they don’t understand data. So, I would say, Data Democratization is a pre-requisite for any type of Self-Service. There are many types, but we will only discuss below.

Types

• Data Self-Service: In this kind of self-service capability, data friendly environment is made available for users, where users are trained on data, users are allowed to question data accuracy and integrity, users are allowed to manipulate and extend the context of data via Data Enrichment process where first-hand data is enriched with second-hand and third-party datasets for better decision making.
• Analytics Self-Service (data citizen): This self-service mostly involves machine learning, data modelling or in other word data science related activities. Data Citizen are major users of this self-service where they run data science or data mining models using low or no code tools. Data Enrichment can also be done using this self-service capability.
• BI Self-Service: As the name explains itself, the users are allowed to connect to any source systems either internal or external, of course based on the access level, and create reports and dashboards on their own. This is majorly to reduce the dependency on IT to create reports for business users. In this era, the paradigm of reports and dashboards creation has moved from IT to Business users.

These terms are the core of the Data Pipeline and are normally confused by each other.

Data Integration is to connect with one or more source systems to pull data whereas Data Blending is to merge all datasets into one viewable and understandable presentation view to answer business KPIs. In other words, Data Integration is the prerequisite process for Data Blending, if you can't get data from source systems, you can't blend it to present management for decision making.

Don't forget either Data Science, Business Intelligence or any kind of Analytics all work toward ONE BIG FAT REALITY, to achieve one or the other Business KPI(s).

Coming back to blending and integration. Don't forget in an organization, we have many systems with different objectives where any ONE kind of data could be residing in different patterns in different systems. For example, a product can be sold via multiple channels like from a shop or an online website or maybe from a mobile application etc. When product data is pulled from all these systems in one Data Warehouse solution to report product selling behavior, it's of utmost importance to gather all product selling transactions from all systems into one system which in our example is a Data Warehouse.

Data Mashup is normally confused with Data Mesh (explained separately), but both are totally different.

Data Mashup comes from the idea of doing self-service on a single source of truth. Dashboard or a Web-Page data point coming from multiple sources in a single view is called Data Mashup. Yes, nowadays, we all do this daily.

We have dashboards with data coming from multiple sources which can be Databases, Data Lakes, social media etc. In the Big Data era, we have graphs, pictures, videos all on one page. At the organization level it’s also called Enterprise Mashup. It is also very useful for Data Analysts to fetch data from different sources, make sense of the data and build meaningful data for business decision-making.

Data Harmonization is normally confused with Data Integration, Data Blending, Data Warehousing, Data Marts etc. Yes, all do the same steps of connecting and bring all required data from all the disparate sources in one central location, but Data Harmonization goes one step ahead.

Data Harmonization standardizes data coming from multiple source systems in one unified format, so it becomes a bridge between data and business requirement. When we say, one unified format, it means data is converted in such a way that it can become accurate for someone.

Data Harmonization can be of many types.

• Harmonizing data in one location for a specific department can be called a Data Mart.
• Defining standard naming conventions, data type, codes coming from multiple sources in one unique format can be Harmonizing data for Master Data Management.
• Gathering Metadata details of all source systems, data lakes, data warehouses, data marts up to reports and dashboards is Harmonizing data for Data Lineage.

Data discovery involves the collection and evaluation of data from various sources and is often used to understand trends and patterns in the data. But understanding trends and patterns cannot be simply done based on data coming from sources such as.

• Data can have Null values.
• Data can have different types of date formats.
• Data can have spelling mistakes.
• Data can have special characters.
• Data can have different codes for the same product e.g., a Glass code can be GLS in one source system and GS in another. So, it's very important to streamline the data sets before presenting them to end users.
• Data can have duplicates.
• And many more....

Data Discovery is different for different roles i.e., for Data Scientist or Data Analyst or Business End User or for C-Level.

• Data Scientists mostly prefer raw data without any data cleaning, cleansing, and scrubbing for Data Dredging, Snooping, p-hacking, and Fishing [there are separate topics on these terms].
• Data analysts mostly prefer data after it’s at least cleaned and cleansed.
• Business End Users and C-Level expect data to be cleaned, cleansed, scrubbed, wrangled, and munged [there are separate topics on these terms] before it reaches them for decision making. Well, a fair expectation.

Monetization is about generating benefit/ revenue/ profit from anything.

‘Data Monetization is about generating revenue from Data.’

In this era, social media is a very dominant channel to interact with customers. Platforms like WhatsApp, Facebook, Twitter, TikTok, Instagram etc. all provide free services to their customers. Have you ever noticed, if you are not paying a single penny to them then how they are generating money and how they are running their business?

The response to the above is by Data Monetization. They sell the data that’s generated by their customers. Having said that, please note, there are regulations that they can’t share personal details of their customers with anyone. But that doesn’t mean they can’t sell how many people are viewing what, how many are buying, how many are spending how much on which product category, from which location they are logged in etc.

For example, on Amazon, customers can buy anything they like. Now at the back end, Amazon must be recording what you are buying and what you are selling. Based on this Information, they can very well analyze which product is more viewed on their portal or what is the buying trend of a particular set of products in a particular season. Like, in December, they can see the trend of products related to Christmas are in great demand. They can sell information like WHICH & HOW MANY PRODUCTS WILL BE IN DEMAND IN COMING DECEMBER based on last few years trend to the product owner to plan their production. This information is GOLD for product owners, they can plan their raw material, work force, timeline and many more activities.

Point-to-Point model (PP) connects all applications with each other e.g., if there are 5 applications then there will be many connections as depicted in below picture.

The Hub-and-Spoke model (SP) creates a new centralized connection points called as Hub and all applications will ONLY connect with centralized ONE Hub.

We can clearly see Hub-and-Spoke model is much better in access management, flexible, cheaper, secure etc. as compared to Point-to-Point model.

Examples of Hub-and-Spoke model are Data Warehouse (dimensional), Data Hub, Data Fabric etc. which serves as single point of truth for all enterprise users.

Key Items

• User Access: SP model can management user access matrix at centralized place as compared to PP model.
• Security: As all users will be connecting to one centralized Hub, risk of security breach will be less in SP model as compared to PP model
• Flexibility: In SP model, Hub is easy to incorporate those new requirements which doesn’t need to implement in source systems.
• Cheaper: In SP model, though Hub implementation requires initial investment but once it’s implemented all new requirements cost will have much cheaper impact as all changes will go in Hub as compared to each application in PP model.

This is a principle rather a process to manipulate data, but it holds a very significant value in terms of managing and securing data.

Most of us mix Critical Data Elements with elements or data points identified during Data Quality exercise. Though those are also critical but when we talk about ‘Critical Data Element’ in Data Strategy, we categorize those elements which can have negative impact on the organization.

Critical Data Element can be any regulatory requirements, any legal or compliance requirements, those data elements which can negatively impact the image of company in general public, those data elements which if are found inaccurate make stakeholders’ loss their confidence on data.

Personal Information of customers is one of the most critical data elements in most of the organization. Credit or Debit Card details in the Banking sector are critical. Location details in Telecommunication are critical etc.

Data Anomalies are about any kind of unusual behavior which can degrade the quality of data. Please note, data quality is not only about dirty data but also about not being able to insert, update and delete required data.

Type of Data Anomalies

• Insertion: If one is unable to enter quality data in database.
• Update: If one is unable to update relevant data or able to update irrelevant data.
• Delete: If one is unable to delete relevant data or able to delete irrelevant data.
• Redundancy: If there are duplicate records in a table.

Data Normalization and database constraints play a vibrant role to make sure data anomalies doesn’t occur. To start with, let’s say there is a table with employee ID, name, department, and salary details. First wrong approach we can noticed i.e., this table is in de-normalized form, by right there should be separate tables for department and salary data using normalization process. Normalization and Denormalization are covered, in detail, in data modelling topic.