Knowledge Hub

Self-Service and Data Enrichment are mostly confused with each other where data enrichment is just one part of one of the types of self-services.

Self-Service is about empowering business users and decision makers to conduct activities by themselves to shorten the end-to-end delivery time. At the same time, it’s too risky to allow people to go and make changes if they don’t understand data. So, I would say, Data Democratization is a pre-requisite for any type of Self-Service. There are many types, but we will only discuss below.

Types

• Data Self-Service: In this kind of self-service capability, data friendly environment is made available for users, where users are trained on data, users are allowed to question data accuracy and integrity, users are allowed to manipulate and extend the context of data via Data Enrichment process where first-hand data is enriched with second-hand and third-party datasets for better decision making.
• Analytics Self-Service (data citizen): This self-service mostly involves machine learning, data modelling or in other word data science related activities. Data Citizen are major users of this self-service where they run data science or data mining models using low or no code tools. Data Enrichment can also be done using this self-service capability.
• BI Self-Service: As the name explains itself, the users are allowed to connect to any source systems either internal or external, of course based on the access level, and create reports and dashboards on their own. This is majorly to reduce the dependency on IT to create reports for business users. In this era, the paradigm of reports and dashboards creation has moved from IT to Business users.

Business Intelligence and Business Analytics both look the same but have different standing in terms of Decision Making.

Business Intelligence has been there for decades with respect to tools and technologies. Business Analytics has also been there but mostly tagged with manual effort by Functional, Business or Financial Consultants.

Howard Dresner coined the modern definition of the term “business intelligence” in 1989, at least in the sense it is typically used in industry today (“end-user access to and analysis of structured content, i.e., data”).

As there weren't a lot of tools available in the market for Analytics, so many companies were making fortunes for this piece of manual effort. Now the question comes, when Business Intelligence tools were there, how come Business Analytics was manual and why Intelligence and Analytics cannot be done using the same tools? For this, one must first understand the differences in how Intelligence and Analytics processing works.

We have covered Business Intelligence, Business Analytics, Data Modelling and Data Cardinality, it's the right time to discuss Data Visualization.

Data Visualization is what is seen by End Users. The Success or Failure of the project is determined based on it. As the phrase goes, 'If you can't draw it, you don't Know it.' And it’s very true. One can spend millions of dollars, can implement a perfect Data Strategy, can have perfect EDW or Data Lake etc. etc. but after all the hard work if the End User can't see what is implementation, if the End User is unable to make use of the data for DSS, if End User will have to go back to their Finance or Technical departments for manual data extraction then all the efforts put-in is nothing more than Waste of Time and Dollars.

Now, there are rules to have good Data Visualization which is relevant to any visualization tools in the market like Tableau, Qlik, Power BI, OBIEE, Cognos and many more.

These terms are the core of the Data Pipeline and are normally confused by each other.

Data Integration is to connect with one or more source systems to pull data whereas Data Blending is to merge all datasets into one viewable and understandable presentation view to answer business KPIs. In other words, Data Integration is the prerequisite process for Data Blending, if you can't get data from source systems, you can't blend it to present management for decision making.

Don't forget either Data Science, Business Intelligence or any kind of Analytics all work toward ONE BIG FAT REALITY, to achieve one or the other Business KPI(s).

Coming back to blending and integration. Don't forget in an organization, we have many systems with different objectives where any ONE kind of data could be residing in different patterns in different systems. For example, a product can be sold via multiple channels like from a shop or an online website or maybe from a mobile application etc. When product data is pulled from all these systems in one Data Warehouse solution to report product selling behavior, it's of utmost importance to gather all product selling transactions from all systems into one system which in our example is a Data Warehouse.

Data Mashup is normally confused with Data Mesh (explained separately), but both are totally different.

Data Mashup comes from the idea of doing self-service on a single source of truth. Dashboard or a Web-Page data point coming from multiple sources in a single view is called Data Mashup. Yes, nowadays, we all do this daily.

We have dashboards with data coming from multiple sources which can be Databases, Data Lakes, social media etc. In the Big Data era, we have graphs, pictures, videos all on one page. At the organization level it’s also called Enterprise Mashup. It is also very useful for Data Analysts to fetch data from different sources, make sense of the data and build meaningful data for business decision-making.

Data Harmonization is normally confused with Data Integration, Data Blending, Data Warehousing, Data Marts etc. Yes, all do the same steps of connecting and bring all required data from all the disparate sources in one central location, but Data Harmonization goes one step ahead.

Data Harmonization standardizes data coming from multiple source systems in one unified format, so it becomes a bridge between data and business requirement. When we say, one unified format, it means data is converted in such a way that it can become accurate for someone.

Data Harmonization can be of many types.

• Harmonizing data in one location for a specific department can be called a Data Mart.
• Defining standard naming conventions, data type, codes coming from multiple sources in one unique format can be Harmonizing data for Master Data Management.
• Gathering Metadata details of all source systems, data lakes, data warehouses, data marts up to reports and dashboards is Harmonizing data for Data Lineage.

Data discovery involves the collection and evaluation of data from various sources and is often used to understand trends and patterns in the data. But understanding trends and patterns cannot be simply done based on data coming from sources such as.

• Data can have Null values.
• Data can have different types of date formats.
• Data can have spelling mistakes.
• Data can have special characters.
• Data can have different codes for the same product e.g., a Glass code can be GLS in one source system and GS in another. So, it's very important to streamline the data sets before presenting them to end users.
• Data can have duplicates.
• And many more....

Data Discovery is different for different roles i.e., for Data Scientist or Data Analyst or Business End User or for C-Level.

• Data Scientists mostly prefer raw data without any data cleaning, cleansing, and scrubbing for Data Dredging, Snooping, p-hacking, and Fishing [there are separate topics on these terms].
• Data analysts mostly prefer data after it’s at least cleaned and cleansed.
• Business End Users and C-Level expect data to be cleaned, cleansed, scrubbed, wrangled, and munged [there are separate topics on these terms] before it reaches them for decision making. Well, a fair expectation.

Heat Map is a graphical representation of objects at any specific location. Nowadays, it is used to identify highly populated area(s) during Covid-19 era. Heat maps make it easy to visualize complex data and understand it briefly.

Now, the question is how Heat Map software knows where the location is. When we open any webpage or any app like Facebook or Twitter etc., via mobile or laptop, our location is picked up. That location is based on Lat and Long (Latitude and Longitude) values. These values are then sent to databases which are then picked up by the Heat Map software for graphical presentation.

• Latitude is the measurement of distance north or south of the Equator. It is measured with 180 imaginary lines that form circles around the Earth east-west, parallel to the Equator.
• Longitude is the measurement east or west of the prime meridian. Longitude is measured by imaginary lines that run around the Earth vertically (up and down) and meet at the North and South Poles. These lines are known as meridians. Each meridian measures one arc degree of longitude. The distance around the Earth measures 360 degrees.

DIKW model

‘Data --> Information --> Knowledge --> Wisdom’

Generally, Data, Information, Knowledge, and Wisdom are arranged in the above sequence. How about considering another angle and trying to arrange these 4 terms in different sequence?

For me Information and Knowledge both comes from Data and Wisdom comes from both Information and Knowledge.

Difference between Data and Information?

As we all know, now a days, Data is everywhere either it is generating from System/ Mobile Applications, Online Websites/ Portals, Videos, Audios, logs etc.

• Is all Data equal to Information? No, it’s NOT.
• Can all Data become Information? Yes, it CAN, based on the nature of the business or requirement.

To answer above, one must first understand what Information is. Information is tightly coupled with what one wants to know. You must have heard this question on many occasions or events or counters etc. like What Information do you need?

Monetization is about generating benefit/ revenue/ profit from anything.

‘Data Monetization is about generating revenue from Data.’

In this era, social media is a very dominant channel to interact with customers. Platforms like WhatsApp, Facebook, Twitter, TikTok, Instagram etc. all provide free services to their customers. Have you ever noticed, if you are not paying a single penny to them then how they are generating money and how they are running their business?

The response to the above is by Data Monetization. They sell the data that’s generated by their customers. Having said that, please note, there are regulations that they can’t share personal details of their customers with anyone. But that doesn’t mean they can’t sell how many people are viewing what, how many are buying, how many are spending how much on which product category, from which location they are logged in etc.

For example, on Amazon, customers can buy anything they like. Now at the back end, Amazon must be recording what you are buying and what you are selling. Based on this Information, they can very well analyze which product is more viewed on their portal or what is the buying trend of a particular set of products in a particular season. Like, in December, they can see the trend of products related to Christmas are in great demand. They can sell information like WHICH & HOW MANY PRODUCTS WILL BE IN DEMAND IN COMING DECEMBER based on last few years trend to the product owner to plan their production. This information is GOLD for product owners, they can plan their raw material, work force, timeline and many more activities.