Knowledge Hub

Metadata Strategy should take place along with project or product strategies, means the earlier the better.

We stated in the Master Data Management term, can Metadata & Master Data reduce your BI footprint by 80%? Yes, it can. For example, if you produce a dashboard/report and describe its Metadata in such a manner that it articulates the entire story around the data points and the KPIs in it then the next BI report will only be generated if the earlier one doesn’t aid the purpose.

In a generic sense we say, Metadata is moreover Data-About-Data, but Metadata is beyond Data-About-Data. It can tell you about business requirements, it can tell you about business goals, it can tell you about Business KPIs etc.

We all have read books either in our student lives or professionally. The Index of the book is Metadata which indicates which page contains which topic. The Acknowledgment page of the book is Metadata as well which tells the overall background, reason, agenda etc., about the book. Metadata can host roles, grants, business processes, terms and conditions, data points and KPI(s) on dashboards and reports, details about databases, tables, columns and rows, the definition of the tables, description of the columns etc. If your Metadata is not in place, you can’t figure out Data Lineage. If your Data Lineage is not placed, your Data will become a Data Swamp, so on…. And again, you are expendable☺. Now is the era of Big Data and Data Lakes, social media, and semi and unstructured data. If one builds a Data Lake without Metadata Management in place, good luck to the organization.

If you have read Edition 2 you know what Data what Metadata is. Now the big question is, are both must-to-have or there can be a possibility to have one only.

Questions:

• Do we need Data when we have Metadata?
  • My response is, No, we don’t.
• In other words, can there be Data with Metadata?
• Again, my response is, No.
  • As one of friend Daniel Lundin said, ‘One person Metadata is Data for another person’.

Exciting topic right….. Yes, it is, as we know size of data is getting huge to humongous lolz 😊.

Let’s decode it…..

Here, we are only referring to Raw Data i.e., which is coming from source systems e.g., OLTP applications or social media or unstructured datasets like video, audio, PDF, email etc., into next datastores e.g., Data Lake or in Data Warehouse etc.

So, when it’s out in the next storage, do we need to keep raw data for life or for certain time e.g., to fulfill regulatory requirements or we can purge it once Metadata has been extracted?

Metadata Management is one of the key components of Data Strategy and Data Management. Is Metadata and Active Metadata two different things. ACTUALLY NO.

Let’s decode it…..

Active Metadata is a new term or jargon surfacing now a days but in-reality, there is no difference between traditional/ passive and active Metadata. Having said that there is difference between how active and passive metadata are generated and managed.

As the name says it all, active metadata is generated automatically and continuously by systems or tools and used to systems approaches like Data Fabric (explained in separate topic).

Data Fabric is the BRAIN where-as, rest of the data components are parts of body in a data solution. BRAIN needs continuous input, BRAIN needs to know everything at the right time rather all the time, dynamically.

A new but very known term, folks from Data World are aware of Metadata but Meta Metadata is something which is just surfaced the Data World and for me it’s again a game changer.

Let’s decode it…..

We all know what Metadata is, it’s Data About Data. Please refer to the Metadata as separate topic in this book.

Let me try to make it as simple as possible with a few examples.

For example, a column i.e., Data Of Birth (DOB) is technical metadata right? Now, when we go to the next level and tag, this DOB is for parents, children, for employees or for insurance policy maker that make the DOB attribute more relevant.

Another example can be Weight column, now value in this field is in numbers or digit but by looking in the value we can’t make it out whether this weight is for a person or an object, and further which object e.g., furniture or something else etc.

Giving more info about Metadata is called Meta Metadata.

Question: Can there be Data with Metadata?

As discussed in a different context, specifically in the comparison between Data and Metadata, my friend Daniel Lundin aptly pointed out that 'One person's Metadata is another person's Data.' This statement underscores the necessity for a profound grasp of the fundamental principles of Data.

Let's delve into the interpretation...

In the discussion on 'Data vs Metadata,' as illustrated in the accompanying image, our attempt was to delineate the distinction between Data and Metadata. To simplify comprehension, we labeled Column Names as Metadata and the actual content as Data.

Let's delve deeper and explore the question: Can there be Data without Metadata? Disclaimer: In my understanding, the answer is NO; it's not possible. Before we delve into further details, let me quickly remind you that traditionally, there are three types of Metadata: Business, Technical, and Operational. The initial image in this discussion illustrates Technical Metadata in the form of Column Names.

Ontology or Knowledge Modelling is the practice of defining a formal and structured representation of knowledge within a domain. It identifies the important concepts, shows how they are related, and defines rules that govern those relationships. The purpose is to create a shared understanding that can be interpreted consistently by both humans and machines.

The main goal of Ontology Modelling is to build a framework that supports semantic clarity, interoperability, and reasoning. Unlike traditional data models that primarily focus on structure and storage, ontologies are designed so that systems can understand meaning and draw logical conclusions from it. This makes them especially valuable in environments where information must be integrated across different systems and interpreted intelligently.

Ontologies go beyond simple definitions of concepts. They define categories, hierarchies, constraints, and rules within a domain. For example, in a healthcare context, an ontology may define that a Patient undergoes a Treatment, and that Treatment can be classified as either a Procedure or a Medication. It may further define that a Medication is a type of Therapeutic Intervention. By establishing such hierarchies and relationships, the ontology provides a structured knowledge framework rather than just isolated definitions.

Metagraph Modelling is an advanced form of graph-based modelling in which nodes and relationships can themselves represent groups, sets, or even entire subgraphs. In other words, it allows relationships to carry their own structure and meaning. This makes it possible to represent complex, multi-level interactions that traditional graph models cannot fully capture.

The primary goal of Metagraph Modelling is to provide a more expressive framework for modelling context, grouping, and higher-order relationships. While a standard graph connects entities through simple relationships, a metagraph can describe relationships about relationships. This added expressiveness is particularly useful in domains where context, provenance, and interaction details are as important as the entities themselves.

To understand the difference, consider a simple graph example. In a loan system, we might represent the fact that a Customer applies for a Loan Application. In a traditional knowledge graph, this would appear as a node for the Customer, a node for the Loan Application, and an edge labeled appliesFor connecting them. This captures the relationship, but it does not fully describe the circumstances under which the application occurred.

Semantic Data Modelling is the practice of designing data structures that are enriched with explicit business meaning so that both humans and machines can interpret and use the data consistently. Unlike purely structural models that define tables, fields, and relationships without deeper context, semantic data models embed meaning directly into the design. They ensure that data is not only structured correctly, but also understood correctly across systems, domains, and even organizational boundaries.

The primary goal of Semantic Data Modelling is to create data models that are self-descriptive and semantically precise. In traditional modelling, a field such as CustomerID may simply be defined as a numeric identifier. In a semantic model, however, that identifier is linked to a formal definition stating that it represents a unique identifier for a person or organization holding an account within a specific business context. The model clarifies what the data means, not just how it is stored. This added semantic layer enables interoperability, governance, and machine-driven reasoning.

Metadata Modelling is the discipline of designing structured frameworks that describe data about data. It defines how business, technical, and operational descriptors are captured, organized, and related so that data assets can be understood, discovered, traced, and governed consistently across the enterprise.

The primary goal of metadata modelling is to establish a structured and standardized approach to cataloguing and managing metadata. By doing so, organizations can improve compliance, strengthen data governance, enhance data quality, and increase trust in analytical and operational systems.

Metadata modelling focuses on descriptors such as:

• Business definitions
• Data ownership and stewardship
• Lineage (upstream and downstream data flow)
• Data source systems
• Refresh frequency
• Sensitivity classification
• Quality rules and thresholds
• Storage location and format

Unlike ontologies, which emphasize semantic meaning and relationships between business concepts, metadata modelling emphasizes the management attributes of data assets. It does not primarily answer “What does a Loan mean?” but rather “Where does the Loan data come from?”, “Who owns it?”, “How often is it refreshed?”, and “Is it sensitive?”

A Data Glossary is a vital step towards mitigating the risk of creating a potential Data Swamp.

Data Glossary vs. Data Dictionary falls under the umbrella of Metadata Management.

• A Data Dictionary is a repository that houses the names, definitions, and attributes of data elements, providing critical contextual information about data.
• A Data Glossary, on the other hand, is a collection of terms that define the key characteristics of your data. These terms are organized in an easy-to-search format to facilitate quick understanding and access.

What Does a Data Glossary Contain?

A Data Glossary includes business terms that provide meaning and context. For instance, a table may have terms like region, sales, quantity, brand, and year. Anyone analysing this table will quickly understand that it relates to sales data, with quantities likely provided brand-wise at a regional level.

A comprehensive Data Glossary can refer to: