Knowledge Hub

Causality Data in the data world refers to information that helps establish a cause-and-effect relationship between different variables or events. In other words, it involves understanding the influence or impact that changes in one variable have on another.

Causality is a fundamental concept in statistics and data analysis, especially when trying to make sense of complex systems or phenomena. Establishing causal relationships is important for drawing meaningful conclusions and making informed decisions. However, it is essential to word that correlation does now no longer imply causation. While two variables may be correlated, it doesn't necessarily mean that one causes the other.

In the context of data science and analytics, establishing causality often involves advanced statistical methods and experimentation.

Here are a few key points related to causality data:

• Experimental Design: Controlled experiments, where researchers manipulate one variable while keeping others constant, are often used to establish causality. This allows for the isolation of the variable of interest and helps draw conclusions about its impact.
• Observational Studies: In cases where controlled experiments are not feasible, observational studies are conducted to analyze existing data. However, establishing causality in observational studies can be more challenging due to potential confounding factors.
• Counterfactuals: Causality often involves comparing what happened to what might have happened in the absence of a particular intervention or change. This comparison is known as a counterfactual analysis.
• Temporal Order: Causality requires that the cause precedes the effect in time. Observing a temporal order between variables is an important aspect of establishing causation.
• Statistical Techniques: Various statistical techniques, such as regression analysis, propensity score matching, and instrumental variable analysis, are employed to analyze data and infer causal relationships.

Inference Data refers to the data produced by ML process of drawing conclusions, making predictions, or generalizing information based on the analysis of a features dataset. It is a crucial step in extracting meaningful insights from data and often involves the application of statistical methods.

Let's decode it.....

Imagine you want to know something about a big group of students, like all the students in your school. Instead of asking every single student (which could be too much work), you ask a smaller group. This smaller group is like a sample. In simpler terms, inference is like making smart guesses about everyone by looking at a few representative friends first. It's a bit like saying, "I can figure out what most people like, by asking a few of them because they're probably a lot like everyone else."

Inference data is a process of drawing meaningful conclusions about a larger dataset based on observed data, and it is a fundamental aspect of statistical reasoning and data analysis in various domains.

If you would like to step into the data domain, the most significant topics anybody must recognize are Metadata, Master & Reference Data Management. If you grasp these areas, one will require 80% less time to orchestrate any Data Solution.

Master Data refers to the core datasets that are essential for business operations and act as the foundation for all other data within an organization. These datasets typically represent key entities or subjects that are critical to the functioning of the business and are used across multiple systems, processes, or applications.

A unique characteristic of Master Data is that it cannot exist independently, it relies on the existence of specific entities or attributes to be meaningful. For example, you cannot have a customer master record without defining key attributes such as the customer’s name, address, and unique identifier (e.g., customer ID). Similarly, a product master record cannot be created without information such as the product name, SKU, or category.

Metadata Strategy should take place along with project or product strategies, means the earlier the better.

We stated in the Master Data Management term, can Metadata & Master Data reduce your BI footprint by 80%? Yes, it can. For example, if you produce a dashboard/report and describe its Metadata in such a manner that it articulates the entire story around the data points and the KPIs in it then the next BI report will only be generated if the earlier one doesn’t aid the purpose.

In a generic sense we say, Metadata is moreover Data-About-Data, but Metadata is beyond Data-About-Data. It can tell you about business requirements, it can tell you about business goals, it can tell you about Business KPIs etc.

We all have read books either in our student lives or professionally. The Index of the book is Metadata which indicates which page contains which topic. The Acknowledgment page of the book is Metadata as well which tells the overall background, reason, agenda etc., about the book. Metadata can host roles, grants, business processes, terms and conditions, data points and KPI(s) on dashboards and reports, details about databases, tables, columns and rows, the definition of the tables, description of the columns etc. If your Metadata is not in place, you can’t figure out Data Lineage. If your Data Lineage is not placed, your Data will become a Data Swamp, so on…. And again, you are expendable☺. Now is the era of Big Data and Data Lakes, social media, and semi and unstructured data. If one builds a Data Lake without Metadata Management in place, good luck to the organization.

If you have read Edition 2 you know what Data what Metadata is. Now the big question is, are both must-to-have or there can be a possibility to have one only.

Questions:

• Do we need Data when we have Metadata?
  • My response is, No, we don’t.
• In other words, can there be Data with Metadata?
• Again, my response is, No.
  • As one of friend Daniel Lundin said, ‘One person Metadata is Data for another person’.

Exciting topic right….. Yes, it is, as we know size of data is getting huge to humongous lolz 😊.

Let’s decode it…..

Here, we are only referring to Raw Data i.e., which is coming from source systems e.g., OLTP applications or social media or unstructured datasets like video, audio, PDF, email etc., into next datastores e.g., Data Lake or in Data Warehouse etc.

So, when it’s out in the next storage, do we need to keep raw data for life or for certain time e.g., to fulfill regulatory requirements or we can purge it once Metadata has been extracted?

Metadata Management is one of the key components of Data Strategy and Data Management. Is Metadata and Active Metadata two different things. ACTUALLY NO.

Let’s decode it…..

Active Metadata is a new term or jargon surfacing now a days but in-reality, there is no difference between traditional/ passive and active Metadata. Having said that there is difference between how active and passive metadata are generated and managed.

As the name says it all, active metadata is generated automatically and continuously by systems or tools and used to systems approaches like Data Fabric (explained in separate topic).

Data Fabric is the BRAIN where-as, rest of the data components are parts of body in a data solution. BRAIN needs continuous input, BRAIN needs to know everything at the right time rather all the time, dynamically.

A new but very known term, folks from Data World are aware of Metadata but Meta Metadata is something which is just surfaced the Data World and for me it’s again a game changer.

Let’s decode it…..

We all know what Metadata is, it’s Data About Data. Please refer to the Metadata as separate topic in this book.

Let me try to make it as simple as possible with a few examples.

For example, a column i.e., Data Of Birth (DOB) is technical metadata right? Now, when we go to the next level and tag, this DOB is for parents, children, for employees or for insurance policy maker that make the DOB attribute more relevant.

Another example can be Weight column, now value in this field is in numbers or digit but by looking in the value we can’t make it out whether this weight is for a person or an object, and further which object e.g., furniture or something else etc.

Giving more info about Metadata is called Meta Metadata.

Question: Can there be Data with Metadata?

As discussed in a different context, specifically in the comparison between Data and Metadata, my friend Daniel Lundin aptly pointed out that 'One person's Metadata is another person's Data.' This statement underscores the necessity for a profound grasp of the fundamental principles of Data.

Let's delve into the interpretation...

In the discussion on 'Data vs Metadata,' as illustrated in the accompanying image, our attempt was to delineate the distinction between Data and Metadata. To simplify comprehension, we labeled Column Names as Metadata and the actual content as Data.

Let's delve deeper and explore the question: Can there be Data without Metadata? Disclaimer: In my understanding, the answer is NO; it's not possible. Before we delve into further details, let me quickly remind you that traditionally, there are three types of Metadata: Business, Technical, and Operational. The initial image in this discussion illustrates Technical Metadata in the form of Column Names.

Data Modelling is the practice of structuring, describing, and organizing information so it reflects real-world entities, relationships, rules, and contexts in a way that supports business operations, analytics, governance, and AI (Artificial Intelligence).

Most of us, are aware of only few Data Modelling techniques, but in reality, there are around 37 of those, shown in below diagram.

Data Modelling Concepts & Techniques

I will try not to go into details in this topic, as we will have separate topics for each of it. First, one must understand the concept of Normalization and Denormalization to get a grasp of data modelling world.

De-normalization: Surely, all of us has worked on Excel. Excel has columns and rows, as do tables in Databases. Consider having employee details in Excel with columns like employee ID, name, mobile number, address, salary, department etc. All this information related to one employee can be stored in one row which is a Denormalization data model.

Flow of all Data Modelling Concepts & Techniques

Data Modelling is a foundational concept in the field of data management and analytics. It involves creating different representations of data structures to support various business and technical requirements.

As mentioned, there are around 37 data modelling technique which can be categorized in 5 layers, mentioned below.

1. Business & Enterprise Modelling
2. Semantic & Information Modelling
3. Core Data Modelling
4. Specialized Techniques
5. Governance & AI Extensions

Let’s see how all data modelling techniques get arranged in these layers.