Knowledge Hub

Metadata Strategy should take place along with project or product strategies, means the earlier the better.

We stated in the Master Data Management term, can Metadata & Master Data reduce your BI footprint by 80%? Yes, it can. For example, if you produce a dashboard/report and describe its Metadata in such a manner that it articulates the entire story around the data points and the KPIs in it then the next BI report will only be generated if the earlier one doesn’t aid the purpose.

In a generic sense we say, Metadata is moreover Data-About-Data, but Metadata is beyond Data-About-Data. It can tell you about business requirements, it can tell you about business goals, it can tell you about Business KPIs etc.

We all have read books either in our student lives or professionally. The Index of the book is Metadata which indicates which page contains which topic. The Acknowledgment page of the book is Metadata as well which tells the overall background, reason, agenda etc., about the book. Metadata can host roles, grants, business processes, terms and conditions, data points and KPI(s) on dashboards and reports, details about databases, tables, columns and rows, the definition of the tables, description of the columns etc. If your Metadata is not in place, you can’t figure out Data Lineage. If your Data Lineage is not placed, your Data will become a Data Swamp, so on…. And again, you are expendable☺. Now is the era of Big Data and Data Lakes, social media, and semi and unstructured data. If one builds a Data Lake without Metadata Management in place, good luck to the organization.

Metadata Management is one of the key components of Data Strategy and Data Management. Is Metadata and Active Metadata two different things. ACTUALLY NO.

Let’s decode it…..

Active Metadata is a new term or jargon surfacing now a days but in-reality, there is no difference between traditional/ passive and active Metadata. Having said that there is difference between how active and passive metadata are generated and managed.

As the name says it all, active metadata is generated automatically and continuously by systems or tools and used to systems approaches like Data Fabric (explained in separate topic).

Data Fabric is the BRAIN where-as, rest of the data components are parts of body in a data solution. BRAIN needs continuous input, BRAIN needs to know everything at the right time rather all the time, dynamically.

While the objective is to imbue data with context, the crucial question is: what type of context are we seeking? Ideally, we aim for a trustworthy and secure context.

Outlined below are 3 high-level steps to guide you in selecting mandatory metadata effectively.

1. Choosing the Right Metadata using the 5W 1H Method: Employ the 5W 1H method (Who, What, When, Where, Why, and How) to meticulously choose metadata that aligns with your specific objectives. This method ensures a comprehensive understanding of the context each metadata element provides.
2. Filtering the metadata with 3 Selection Criteria: Streamline your selection further by applying three essential criteria to the metadata options:

• For Everyone: Choose metadata that anyone can understand, not tied to specific organizations or industries. This makes data easy to use, keeps it relevant, and encourages more people to get involved.
• Anyone Can Create: Pick metadata that anyone familiar with the organization's data can create. No need for technical or legal expertise. This makes the process open to everyone, speeds things up, and promotes collaboration.
• Follow Standards: Ensure your metadata follows universal standards. This keeps things consistent internally and aligns with best practices in the industry, making it easy to share data with others.

Semantic Data Modelling is the practice of designing data structures that are enriched with explicit business meaning so that both humans and machines can interpret and use the data consistently. Unlike purely structural models that define tables, fields, and relationships without deeper context, semantic data models embed meaning directly into the design. They ensure that data is not only structured correctly, but also understood correctly across systems, domains, and even organizational boundaries.

The primary goal of Semantic Data Modelling is to create data models that are self-descriptive and semantically precise. In traditional modelling, a field such as CustomerID may simply be defined as a numeric identifier. In a semantic model, however, that identifier is linked to a formal definition stating that it represents a unique identifier for a person or organization holding an account within a specific business context. The model clarifies what the data means, not just how it is stored. This added semantic layer enables interoperability, governance, and machine-driven reasoning.

We discussed the Data Dictionary and the Data or Business Glossary. What is Data Catalog?

‘Data Catalog comes under Metadata Management.’

Although all three Data Dictionaries, Data or Business Glossary and Data Catalog look similar, it's very important to understand the difference.

Data Dictionary stores metadata details about the solution and Data or Business Glossary is the collection of all business terms about the solution whereas Data Catalog jointly refers to Dictionary and Glossary like which business term is referring to which table or dashboard etc. In other words, Catalog gives a complete picture of both Dictionary and Glossary.

Let's take an example of a Bank Account Number (BAN).

• Data Glossary: BAN is the unique identifier for a customer across all banking applications whether it’s HR, Finance, Marketing etc.
• Data Dictionary: BAN is a unique identifier with field type as string, field length as 10, with two times # in it etc., metadata detail.
• Data Catalog: It tells the whole story i.e., BAN is the unique identifier for a customer across all banking applications and it is stored in tables i.e., x, y, and z, it is in which schema and database. And which department owns which databases, tables, dashboards, and reports etc.

We have been referring to three terms i.e., Master Data Management, Metadata Management and Data Lineage, in previous topics. Master Data and Metadata Management have already been covered. Now it’s time to decode Data Lineage which is THE most important FACTOR to save Data Lake from becoming a Data Swamp. Efficient Data Lineage can’t be built without Metadata in place.

I won’t go into details for Data Swamp, there is a separate topic on it. But any data showing in your BI Solution, and you are unable to track it back to its source means you don’t have Data Lineage and you are heading towards disaster as if not now but in future you will have no idea where the data is coming from and where it’s been used.

Data Lineage is the process of tagging all the touchpoints i.e., databases, tables, columns, business roles etc., from where data has been traveled from to BI solution or downstream systems. It also shows all the ETL(s) it went through, all the data enrichments, transformations, conversions, aggregations, and all the new measures created along the way. This helps businesses to trust the completeness and accuracy of the data for their decision-making. Data Lineage provides an audit trail of data lifecycle throughout the enterprise and assists users to understand UAM (User Access Metric).

Data Provenance and Data Lineage. Logically both do the same thing but at different levels of granularity.

Data Lineage is about knowing the Data Lifecycle but with respect to databases, tables, columns, business rule/ ETL rule etc. For example, data comes from which database, table, column and went into which database, table, column with what business rules incorporated during transformation process.

Data Provenance is at a higher level, it’s about Data Lifecycle with respect to inputs, entities, systems, and processes. For example, data comes in which format, from which source system, its related to which entity, what is the process used like real-time processing, batch processing or event based etc., any analytical modelling done, any data enrichment took place, any data cleaning, data quality rules, metadata management, master & reference data management etc., took place during the data lifecycle.

These two terms are used in Machine Learning for Data Mining and Data Science models. As we all know, within Machine Learning, there are Supervised and Unsupervised Learning methods.

Data Classification/Categorization=Supervised Learning

Data Clustering=Unsupervised Learnings

Both these terms are used for patterns identification before complex algorithms come into action.

Classification is done based on the uniqueness of the data where e.g., all animals are tagged separately whereas Clustering is done based on the characteristics e.g., 1) Animals with 2 legs 2) Animals with 4 legs.

• Data Classification uses labeled data whereas Data Clustering uses unlabeled data.
• The most popular Data Classification algorithms in data mining are the K-Nearest Neighbor and decision tree algorithms.
• The two common Data Clustering algorithms in data mining are K-means clustering and hierarchical clustering.
• Data Classification output is known.
• Data Clustering out is unknown.
• Training data is provided for Data Classification.
• No training data is provided for Data Clustering.

Data Annotation and Data Labelling works together to prepare dataset for Machine Learning Models in Data Science practice.

‘Data Annotation is the technique of doing Data Labelling.’

Data Annotation is a complete process with many steps to curate, whereas Data Labelling is ONE of the many steps in it.

Data Labelling is about tagging the data for training e.g., machine learning models whereas Data Annotation is the technique for tagging the data so ML models can reuse it as well. Data Labelling tags are used by ML models for future pattern recognitions whereas Data Annotation is about recognizing the data via a process.

Just like Data Labelling, Data Annotation is also for supervised machine learning to prepare data as an input and to store it for future reference. Not to mention, there are STILL many organizations doing manual Data Labelling. Data Annotation is about the process so Data Labelling can be automated for accuracy, eliminating human intervention. Bad Data Labelling can impact the outcome of applications.

Data discovery involves the collection and evaluation of data from various sources and is often used to understand trends and patterns in the data. But understanding trends and patterns cannot be simply done based on data coming from sources such as.

• Data can have Null values.
• Data can have different types of date formats.
• Data can have spelling mistakes.
• Data can have special characters.
• Data can have different codes for the same product e.g., a Glass code can be GLS in one source system and GS in another. So, it's very important to streamline the data sets before presenting them to end users.
• Data can have duplicates.
• And many more....

Data Discovery is different for different roles i.e., for Data Scientist or Data Analyst or Business End User or for C-Level.

• Data Scientists mostly prefer raw data without any data cleaning, cleansing, and scrubbing for Data Dredging, Snooping, p-hacking, and Fishing [there are separate topics on these terms].
• Data analysts mostly prefer data after it’s at least cleaned and cleansed.
• Business End Users and C-Level expect data to be cleaned, cleansed, scrubbed, wrangled, and munged [there are separate topics on these terms] before it reaches them for decision making. Well, a fair expectation.