Knowledge Hub

Metadata Strategy should take place along with project or product strategies, means the earlier the better.

We stated in the Master Data Management term, can Metadata & Master Data reduce your BI footprint by 80%? Yes, it can. For example, if you produce a dashboard/report and describe its Metadata in such a manner that it articulates the entire story around the data points and the KPIs in it then the next BI report will only be generated if the earlier one doesn’t aid the purpose.

In a generic sense we say, Metadata is moreover Data-About-Data, but Metadata is beyond Data-About-Data. It can tell you about business requirements, it can tell you about business goals, it can tell you about Business KPIs etc.

We all have read books either in our student lives or professionally. The Index of the book is Metadata which indicates which page contains which topic. The Acknowledgment page of the book is Metadata as well which tells the overall background, reason, agenda etc., about the book. Metadata can host roles, grants, business processes, terms and conditions, data points and KPI(s) on dashboards and reports, details about databases, tables, columns and rows, the definition of the tables, description of the columns etc. If your Metadata is not in place, you can’t figure out Data Lineage. If your Data Lineage is not placed, your Data will become a Data Swamp, so on…. And again, you are expendable☺. Now is the era of Big Data and Data Lakes, social media, and semi and unstructured data. If one builds a Data Lake without Metadata Management in place, good luck to the organization.

If you have read Edition 2 you know what Data what Metadata is. Now the big question is, are both must-to-have or there can be a possibility to have one only.

Questions:

• Do we need Data when we have Metadata?
  • My response is, No, we don’t.
• In other words, can there be Data with Metadata?
• Again, my response is, No.
  • As one of friend Daniel Lundin said, ‘One person Metadata is Data for another person’.

Exciting topic right….. Yes, it is, as we know size of data is getting huge to humongous lolz 😊.

Let’s decode it…..

Here, we are only referring to Raw Data i.e., which is coming from source systems e.g., OLTP applications or social media or unstructured datasets like video, audio, PDF, email etc., into next datastores e.g., Data Lake or in Data Warehouse etc.

So, when it’s out in the next storage, do we need to keep raw data for life or for certain time e.g., to fulfill regulatory requirements or we can purge it once Metadata has been extracted?

Metadata Management is one of the key components of Data Strategy and Data Management. Is Metadata and Active Metadata two different things. ACTUALLY NO.

Let’s decode it…..

Active Metadata is a new term or jargon surfacing now a days but in-reality, there is no difference between traditional/ passive and active Metadata. Having said that there is difference between how active and passive metadata are generated and managed.

As the name says it all, active metadata is generated automatically and continuously by systems or tools and used to systems approaches like Data Fabric (explained in separate topic).

Data Fabric is the BRAIN where-as, rest of the data components are parts of body in a data solution. BRAIN needs continuous input, BRAIN needs to know everything at the right time rather all the time, dynamically.

A new but very known term, folks from Data World are aware of Metadata but Meta Metadata is something which is just surfaced the Data World and for me it’s again a game changer.

Let’s decode it…..

We all know what Metadata is, it’s Data About Data. Please refer to the Metadata as separate topic in this book.

Let me try to make it as simple as possible with a few examples.

For example, a column i.e., Data Of Birth (DOB) is technical metadata right? Now, when we go to the next level and tag, this DOB is for parents, children, for employees or for insurance policy maker that make the DOB attribute more relevant.

Another example can be Weight column, now value in this field is in numbers or digit but by looking in the value we can’t make it out whether this weight is for a person or an object, and further which object e.g., furniture or something else etc.

Giving more info about Metadata is called Meta Metadata.

Question: Can there be Data with Metadata?

As discussed in a different context, specifically in the comparison between Data and Metadata, my friend Daniel Lundin aptly pointed out that 'One person's Metadata is another person's Data.' This statement underscores the necessity for a profound grasp of the fundamental principles of Data.

Let's delve into the interpretation...

In the discussion on 'Data vs Metadata,' as illustrated in the accompanying image, our attempt was to delineate the distinction between Data and Metadata. To simplify comprehension, we labeled Column Names as Metadata and the actual content as Data.

Let's delve deeper and explore the question: Can there be Data without Metadata? Disclaimer: In my understanding, the answer is NO; it's not possible. Before we delve into further details, let me quickly remind you that traditionally, there are three types of Metadata: Business, Technical, and Operational. The initial image in this discussion illustrates Technical Metadata in the form of Column Names.

While the objective is to imbue data with context, the crucial question is: what type of context are we seeking? Ideally, we aim for a trustworthy and secure context.

Outlined below are 3 high-level steps to guide you in selecting mandatory metadata effectively.

1. Choosing the Right Metadata using the 5W 1H Method: Employ the 5W 1H method (Who, What, When, Where, Why, and How) to meticulously choose metadata that aligns with your specific objectives. This method ensures a comprehensive understanding of the context each metadata element provides.
2. Filtering the metadata with 3 Selection Criteria: Streamline your selection further by applying three essential criteria to the metadata options:

• For Everyone: Choose metadata that anyone can understand, not tied to specific organizations or industries. This makes data easy to use, keeps it relevant, and encourages more people to get involved.
• Anyone Can Create: Pick metadata that anyone familiar with the organization's data can create. No need for technical or legal expertise. This makes the process open to everyone, speeds things up, and promotes collaboration.
• Follow Standards: Ensure your metadata follows universal standards. This keeps things consistent internally and aligns with best practices in the industry, making it easy to share data with others.

Data Modelling is the practice of structuring, describing, and organizing information so it reflects real-world entities, relationships, rules, and contexts in a way that supports business operations, analytics, governance, and AI (Artificial Intelligence).

Most of us, are aware of only few Data Modelling techniques, but in reality, there are around 37 of those, shown in below diagram.

Data Modelling Concepts & Techniques

I will try not to go into details in this topic, as we will have separate topics for each of it. First, one must understand the concept of Normalization and Denormalization to get a grasp of data modelling world.

De-normalization: Surely, all of us has worked on Excel. Excel has columns and rows, as do tables in Databases. Consider having employee details in Excel with columns like employee ID, name, mobile number, address, salary, department etc. All this information related to one employee can be stored in one row which is a Denormalization data model.

Flow of all Data Modelling Concepts & Techniques

Data Modelling is a foundational concept in the field of data management and analytics. It involves creating different representations of data structures to support various business and technical requirements.

As mentioned, there are around 37 data modelling technique which can be categorized in 5 layers, mentioned below.

1. Business & Enterprise Modelling
2. Semantic & Information Modelling
3. Core Data Modelling
4. Specialized Techniques
5. Governance & AI Extensions

Let’s see how all data modelling techniques get arranged in these layers.

Business Modelling is the practice of understanding how an organization works as a whole. It explains how value is created, how work flows through the organization, who is involved at each step, and how decisions are made. At this stage, the focus is not on data, databases, or systems, but on the business itself. The aim is to clearly describe what the organization does and how it achieves its objectives.

The main goal of Business Modelling is to provide a complete and holistic view of the business. When this view is clear, it becomes much easier to align daily operations, management decisions, and supporting systems with the organization’s strategic goals. Without this shared understanding, systems and data solutions are often built in isolation and fail to support real business needs.

Business Modelling concentrates on the “what” and the “how” of the business. It looks at business processes, roles, responsibilities, rules, and policies. It helps answer questions such as: What activities are performed? Who performs them? What decisions are taken, and based on what rules? Because of this, Business Modelling is usually the first step before moving into information modelling and data modelling. It establishes where data is created, how it flows, and how it is ultimately used.

Process Modelling, often referred to as Business Process Modelling (BPM), is the practice of describing how work is carried out step by step inside an organization. It visually represents workflows, activities, and decision points so that everyone can clearly understand how a process starts, how it moves forward, and how it ends. Unlike Business Modelling, which explains the business at a high level, Process Modelling focuses on the actual flow of work.

The main goal of Process Modelling is to analyse and improve business processes. By clearly documenting each step, organizations can identify delays, unnecessary work, repeated tasks, or unclear responsibilities. Once these issues are visible, processes can be standardized, improved, or automated to achieve better efficiency and consistency while staying aligned with organizational objectives.

Process Modelling provides a structured way to document how work moves across people, systems, and departments. It shows who performs each task, when decisions are made, and how work is handed over from one role to another. Because of this, Process Modelling often acts as a bridge between business requirements and system design. It is commonly used as a foundation for automation, compliance checks, and process improvement initiatives.