Knowledge Hub

If you have read Edition 2 you know what Data what Metadata is. Now the big question is, are both must-to-have or there can be a possibility to have one only.

Questions:

• Do we need Data when we have Metadata?
  • My response is, No, we don’t.
• In other words, can there be Data with Metadata?
• Again, my response is, No.
  • As one of friend Daniel Lundin said, ‘One person Metadata is Data for another person’.

Exciting topic right….. Yes, it is, as we know size of data is getting huge to humongous lolz 😊.

Let’s decode it…..

Here, we are only referring to Raw Data i.e., which is coming from source systems e.g., OLTP applications or social media or unstructured datasets like video, audio, PDF, email etc., into next datastores e.g., Data Lake or in Data Warehouse etc.

So, when it’s out in the next storage, do we need to keep raw data for life or for certain time e.g., to fulfill regulatory requirements or we can purge it once Metadata has been extracted?

Question: Can there be Data with Metadata?

As discussed in a different context, specifically in the comparison between Data and Metadata, my friend Daniel Lundin aptly pointed out that 'One person's Metadata is another person's Data.' This statement underscores the necessity for a profound grasp of the fundamental principles of Data.

Let's delve into the interpretation...

In the discussion on 'Data vs Metadata,' as illustrated in the accompanying image, our attempt was to delineate the distinction between Data and Metadata. To simplify comprehension, we labeled Column Names as Metadata and the actual content as Data.

Let's delve deeper and explore the question: Can there be Data without Metadata? Disclaimer: In my understanding, the answer is NO; it's not possible. Before we delve into further details, let me quickly remind you that traditionally, there are three types of Metadata: Business, Technical, and Operational. The initial image in this discussion illustrates Technical Metadata in the form of Column Names.

A Data Glossary is a vital step towards mitigating the risk of creating a potential Data Swamp.

Data Glossary vs. Data Dictionary falls under the umbrella of Metadata Management.

• A Data Dictionary is a repository that houses the names, definitions, and attributes of data elements, providing critical contextual information about data.
• A Data Glossary, on the other hand, is a collection of terms that define the key characteristics of your data. These terms are organized in an easy-to-search format to facilitate quick understanding and access.

What Does a Data Glossary Contain?

A Data Glossary includes business terms that provide meaning and context. For instance, a table may have terms like region, sales, quantity, brand, and year. Anyone analysing this table will quickly understand that it relates to sales data, with quantities likely provided brand-wise at a regional level.

A comprehensive Data Glossary can refer to:

A Data Dictionary is the next step towards mitigating the Risk of a potential Data Swamp.

‘Data Dictionary comes under Metadata Management.’

In the IT World, we all know that new products and solutions keep coming year by year, so it's very important to capture what has been implemented. A Data Dictionary is one of the most important items for a system's impact analysis as it gives a clear picture of all the elements in the database.

With the introduction of Big Data where semi and unstructured datasets have come onto surfaces like video, audio, logs, emails, pdf & social media etc., so if one doesn't have a Data Dictionary in place then all future implementations will take 2x-3x more time.

Data Dictionary stores metadata information about databases like.

• List of all the objects and when each was created or updated.
• Details like field name, description, data type, length, indexes, primary & foreign keys etc.
• Relationships between tables, schema(s) and databases etc.
• Number of records in each table

This is a very confusing topic as both Categorization and Classification are used interchangeably.

Both Categorization and Classification are Metadata and are used for Data Governance as well.

Categorization is based on the attributes and properties of entities in dataset whereas Classification is based on the business requirements.

As you can refer to the image, data has been categorized in passport, health record, financial record, job posting etc., and at the same time these categories are classified as Medium Risk.

‘Both these terms fall under Supervised Learning in Data Science domain.’

For example, there are grandparents, parent, children, cousins, servants, gardener, machinic, car painter etc., all these falls under categories but there can be further classifications i.e., family members and non-family members.

Categories: Grandparents, Parent, Children, Cousins, Gardener, Machinic, Car painters