Knowledge Hub

Do We Really Need Data Modelling?

In recent years, a recurring question has surfaced in data conversations: Do we really need data modelling anymore? With cloud-native platforms, data lakes, AI pipelines, and schema-on-read architectures dominating the narrative, some assume modelling is optional, or even obsolete.

The reality is far simpler. There is no such thing as data without a model. Whether designed consciously or not, every system that stores data follows a structure. That structure, explicit or implicit, is a data model.

When we build operational applications, we design backend databases. These typically follow normalization principles, often in Third Normal Form (3NF). Customer data is stored separately from orders, orders from payments, and relationships are maintained through keys and constraints. This structure prevents redundancy, preserves integrity, and ensures consistency. That is data modelling in action.

Entropy term is normally used in Data Science domain.

• Where finding unexpected outcomes is the aim
• Where surprises are welcomed
• Where results are analyzed based on probability ratio, the lower the probability the better it is.
• Where informative information is something which wasn’t known

Types

• High Entropy means, more surprises, more unexpected values are considered more informative.
• Low Entropy means, less surprises, less unexpected values are considered less informative.

‘Entropy is also called as Extreme Disorder of values.’

Referring to the image, we can see at the starting point all the signs are MINUS, then in the middle there are 50/50 signs of PLUS & MINUS and right at the end all the signs are PLUS. At those extremes left, middle and right corners, the Entropy is at the lowest, so no surprises are expected so nothing much for Data Scientists to predict right. But in 2^nd, 4^th, and 5^th circles, it’s difficult to say how many PLUSES(s) and MINUS(s), if values are not visible, at this points Entropy is at the highest.

Data Dividend is something everyone must know either IT Professional or End User.

We all know what a Dividend is in the Stock Market. It's when a user buys a Stock of a particular company and keeps it for a certain time, the company pays the user a certain amount in Quarterly or Annual basis. The reason is the company is keeping users' money for a particular duration. At the backend, the company is investing that money in its business to generate revenue. Then a certain percentage of that revenue is shared with the user in the shape of a Dividend.

This era is to generate customer data either via mobile applications or via social media portals like Facebook, Twitter, Instagram, Amazon, Alibaba, etc. Users use these applications and, in a way, give their DATA instead of MONEY like their buying/ selling behavior, their search trend, their family/ friends interactions, etc. to above-mentioned companies which then use that DATA for marketing & advertisement, for selling services, to production houses, to industries, to inventory houses, etc. In this cycle, all these companies are making Fortunes, and nothing has been shared with the ENTITY i.e., end-users who are the core participants for the data generation.

‘Intuition is the ability to understand something without any need for conscious reasoning.’

Intuition is the ability to acquire knowledge without conscious reasoning. It includes but is not limited to direct access to unconscious knowledge, and the ability to understand something instinctively, without any need for conscious reasoning.

Now, how does it relate to Data? I would say, Data Intuition is one of the most important capabilities any data analyst, data citizen, data scientist and management should have. Yes, decisions can be taken based on data, but one should not take a decision, if his/ her intuition is not aligned.

For example, there are sales of 200k drinking bottles per month, expected from a particular area. But suddenly, 500k bottles were sold, NOW your data intuition should shake you about the result. Now you can start asking WHY’s, till you have all the evidence of the sales. It could be a fake result just to show higher sales to get a bonus 😊.

To understand Data Algorithms, let's quickly know what an algorithm is. It is a procedure consisting of multiple steps with a set of instructions for a desired result. It is independent of any language, in other words, an algorithm can be implemented in any language.

Algorithm Characteristics

• Unambiguous: It should be clear and unambiguous. Each of its steps, and its inputs/outputs should be clear and must lead to only one meaning.
• Input: It should have 0 or more well-defined inputs.
• Output: It should have 1 or more well-defined outputs and should match the desired output.
• Finiteness: It must terminate after a finite number of steps.
• Feasibility: It should be feasible with the available resources.
• Independent: It should have step-by-step directions, which should be independent of any programming code.

Now from a data point-of-view, how the algorithm behaves.

• Search: Algorithm to search an item in a data structure.
• Sort: Algorithm to sort items in a certain order.
• Insert: Algorithm to insert an item in a data structure.
• Update: Algorithm to update an existing item in a data structure.
• Delete: Algorithm to delete an existing item from a data structure.

DIKW model

‘Data --> Information --> Knowledge --> Wisdom’

Generally, Data, Information, Knowledge, and Wisdom are arranged in the above sequence. How about considering another angle and trying to arrange these 4 terms in different sequence?

For me Information and Knowledge both comes from Data and Wisdom comes from both Information and Knowledge.

Difference between Data and Information?

As we all know, now a days, Data is everywhere either it is generating from System/ Mobile Applications, Online Websites/ Portals, Videos, Audios, logs etc.

• Is all Data equal to Information? No, it’s NOT.
• Can all Data become Information? Yes, it CAN, based on the nature of the business or requirement.

To answer above, one must first understand what Information is. Information is tightly coupled with what one wants to know. You must have heard this question on many occasions or events or counters etc. like What Information do you need?

Data Surfing is always confused with Data Analysis. There is a very thin line between Data Analysts and Data Surfers.

Data Surfing is normally done by non-technical people like management who doesn’t know how to write any kind of code to access data at database level.

Data Surfers normally stays at the level of Dashboards and Reports where they just click here and there to explore data for required information.

Business Intelligence (BI) tools came into being to cater Data Surfers World where data was required to be presented for non-technical people who could view data for decision making.

Era(s) for Data Surfers

• Static Reports: In the beginning, business users used to share their data requirements with IT who then used to create dashboards and reports. Those dashboards/ reports and data was not changeable, it was in paper format or in excel format or in text format etc., means after Go-Live if business want one more column or if latest data was required then the whole development cycle had to restart from scratch.
• Dynamic Reports: In the second era, BI tools came-in to cater for this gap. To start with, dashboards/ reports were still not changeable, but data was directly connected with backend databases, so data was always latest at front-end. This was a huge jump, as now, to view new data developer don’t have to recreate front-end again and again.
• Adhoc Reports: Latest era is where dashboards/ reports are also changeable. Now, end users and management can drag and drop tables and columns in dashboards/ reports.

Data Contextualization has emerged as a critical factor in dealing with and summarizing vast, rapidly changing, and diverse unstructured datasets. Let's break it down.

We all understand the concept of context. For instance, when you're about to commence a meeting with management, you begin by acknowledging them and stating the purpose of the meeting. This establishes the context, ensuring that the audience knows what to expect and how to interpret the upcoming presentation.

Similarly, in the realm of data, Data Contextualization has become a fundamental process, especially when extracting insights from unstructured data. Unstructured data typically contains a substantial amount of information that is irrelevant to decision-making. The relevance of a data element may vary depending on the specific decision being made.

Here's why Data Contextualization is essential:

• Filtering Irrelevant Data: It eliminates irrelevant data, reducing the overall volume of data. This, in turn, simplifies data processing, cleaning, transformation, and maintenance.
• Enhancing Decision-Making: By contextualizing data, you can combine external data with internal data, providing a more comprehensive foundation for decision-making.
• Providing Clarity: It offers a clear rationale for the data, making it easier to comprehend and analyze.
• Adding Points of Interest: Data Contextualization allows for the inclusion of relevant insights or points of interest about the data, facilitating better understanding.

Data Literacy Assessment involves evaluating an individual's ability to understand, interpret, and work with data effectively. It measures how well someone can read, analyze, and draw insights from data.

Let's elaborate on each aspect of data literacy assessment:

• Data Understanding: This aspect assesses an individual's ability to comprehend and make sense of the data presented. It involves understanding the data structure, variables, and context. For example, imagine you have a table with information about the number of apples sold each day at a fruit stand. A data literacy assessment might ask you to look at the table and answer questions like, "On which day were the most apples sold?" This tests your ability to understand the data presented.
• Data Interpretation: It evaluates how well someone can analyze and draw meaningful insights from the data. It includes recognizing patterns, trends, and relationships within the dataset. For example, you are given a chart showing the temperature variations over a week. A data literacy assessment could ask, "What can you conclude about the weather during this week?" This evaluates your capacity to interpret data trends and draw meaningful conclusions.
• Data Presentation: It focuses on the skill of effectively communicating data findings. It involves selecting appropriate visualizations or formats to convey information clearly to a specific audience. For example, suppose you are asked to create a simple bar chart using data about the number of books read by different students. This assesses your ability to choose an appropriate way to present data visually.
• Data Analysis: It evaluates an individual's capability to process and manipulate data to uncover patterns, correlations, and trends. It entails making use of statistical and computational strategies to derive insights. For example, you are provided with a data set on students' test scores and attendance. A data literacy assessment might request you to identify if there's a correlation between attendance and academic performance. This tests your ability to analyze data for patterns and relationships.
• Problem-Solving with Data: This aspect assesses the ability to use data to solve real-world problems. It involves applying data-driven approaches to make informed decisions and address challenges. For example, given data on the growth of plants under different conditions, a data literacy assessment could ask, "Which condition is most favorable for plant growth, and why?" This evaluates your capability to apply information to resolve real-international problems.
• Data Ethics: It examines an individual's understanding of ethical considerations related to data. It involves recognizing and addressing issues such as privacy, consent, and fairness in the handling of data. For example, you are presented with a scenario where a company is collecting customer data. A data literacy assessment might ask, "What ethical considerations should the company take into account when handling this data?" This assesses your understanding of ethical implications related to data.
• Data Communication: It assesses how well someone can convey data insights to others. It entails growing reports, visualizations, or narratives that efficiently talk findings to each technical and non-technical audience. For example, you are given a dataset on pollution levels in a city. A data literacy assessment could ask you to prepare a short report for a non-technical audience, explaining the findings and potential implications. This tests your ability to communicate data insights effectively.
• Critical Thinking with Data: Critical thinking with data evaluates the ability to approach data with a questioning mindset. It involves considering alternative perspectives, asking relevant questions, and identifying potential biases in the data. For example, you receive data about the popularity of different sports in a school. A data literacy assessment might ask, "What other information would you need to determine the factors influencing students' sports preferences?" This assesses your analytical reasoning abilities in consuming data.

In the data world, "Digitalization" and "Digitization" are related concepts, but they have distinct meanings and implications.

Let's decode the difference between the two.

Digitization: It refers to the process of converting analog information into digital format. It involves taking physical or analog data, such as text, images, or sound, and transforming it into digital form, typically represented as binary code (0s and 1s) that computers can understand. For example, scanning a paper document and converting it into a digital PDF file is an example of digitization. The original document exists in analog (paper) form, and digitization converts it into a digital representation.

Digitalization: It is a broader concept that involves the use of digital technologies to transform business processes, activities, or models. It goes beyond converting analog data to digital format and encompasses the integration of digital technologies into various aspects of an organization or system to enhance efficiency, functionality, or value.