Knowledge Hub

A Data Glossary is a vital step towards mitigating the risk of creating a potential Data Swamp.

Data Glossary vs. Data Dictionary falls under the umbrella of Metadata Management.

• A Data Dictionary is a repository that houses the names, definitions, and attributes of data elements, providing critical contextual information about data.
• A Data Glossary, on the other hand, is a collection of terms that define the key characteristics of your data. These terms are organized in an easy-to-search format to facilitate quick understanding and access.

What Does a Data Glossary Contain?

A Data Glossary includes business terms that provide meaning and context. For instance, a table may have terms like region, sales, quantity, brand, and year. Anyone analysing this table will quickly understand that it relates to sales data, with quantities likely provided brand-wise at a regional level.

A comprehensive Data Glossary can refer to:

A Data Dictionary is the next step towards mitigating the Risk of a potential Data Swamp.

‘Data Dictionary comes under Metadata Management.’

In the IT World, we all know that new products and solutions keep coming year by year, so it's very important to capture what has been implemented. A Data Dictionary is one of the most important items for a system's impact analysis as it gives a clear picture of all the elements in the database.

With the introduction of Big Data where semi and unstructured datasets have come onto surfaces like video, audio, logs, emails, pdf & social media etc., so if one doesn't have a Data Dictionary in place then all future implementations will take 2x-3x more time.

Data Dictionary stores metadata information about databases like.

• List of all the objects and when each was created or updated.
• Details like field name, description, data type, length, indexes, primary & foreign keys etc.
• Relationships between tables, schema(s) and databases etc.
• Number of records in each table

We discussed the Data Dictionary and the Data or Business Glossary. What is Data Catalog?

‘Data Catalog comes under Metadata Management.’

Although all three Data Dictionaries, Data or Business Glossary and Data Catalog look similar, it's very important to understand the difference.

Data Dictionary stores metadata details about the solution and Data or Business Glossary is the collection of all business terms about the solution whereas Data Catalog jointly refers to Dictionary and Glossary like which business term is referring to which table or dashboard etc. In other words, Catalog gives a complete picture of both Dictionary and Glossary.

Let's take an example of a Bank Account Number (BAN).

• Data Glossary: BAN is the unique identifier for a customer across all banking applications whether it’s HR, Finance, Marketing etc.
• Data Dictionary: BAN is a unique identifier with field type as string, field length as 10, with two times # in it etc., metadata detail.
• Data Catalog: It tells the whole story i.e., BAN is the unique identifier for a customer across all banking applications and it is stored in tables i.e., x, y, and z, it is in which schema and database. And which department owns which databases, tables, dashboards, and reports etc.

We have been referring to three terms i.e., Master Data Management, Metadata Management and Data Lineage, in previous topics. Master Data and Metadata Management have already been covered. Now it’s time to decode Data Lineage which is THE most important FACTOR to save Data Lake from becoming a Data Swamp. Efficient Data Lineage can’t be built without Metadata in place.

I won’t go into details for Data Swamp, there is a separate topic on it. But any data showing in your BI Solution, and you are unable to track it back to its source means you don’t have Data Lineage and you are heading towards disaster as if not now but in future you will have no idea where the data is coming from and where it’s been used.

Data Lineage is the process of tagging all the touchpoints i.e., databases, tables, columns, business roles etc., from where data has been traveled from to BI solution or downstream systems. It also shows all the ETL(s) it went through, all the data enrichments, transformations, conversions, aggregations, and all the new measures created along the way. This helps businesses to trust the completeness and accuracy of the data for their decision-making. Data Lineage provides an audit trail of data lifecycle throughout the enterprise and assists users to understand UAM (User Access Metric).

Data Provenance and Data Lineage. Logically both do the same thing but at different levels of granularity.

Data Lineage is about knowing the Data Lifecycle but with respect to databases, tables, columns, business rule/ ETL rule etc. For example, data comes from which database, table, column and went into which database, table, column with what business rules incorporated during transformation process.

Data Provenance is at a higher level, it’s about Data Lifecycle with respect to inputs, entities, systems, and processes. For example, data comes in which format, from which source system, its related to which entity, what is the process used like real-time processing, batch processing or event based etc., any analytical modelling done, any data enrichment took place, any data cleaning, data quality rules, metadata management, master & reference data management etc., took place during the data lifecycle.

These two terms are used in Machine Learning for Data Mining and Data Science models. As we all know, within Machine Learning, there are Supervised and Unsupervised Learning methods.

Data Classification/Categorization=Supervised Learning

Data Clustering=Unsupervised Learnings

Both these terms are used for patterns identification before complex algorithms come into action.

Classification is done based on the uniqueness of the data where e.g., all animals are tagged separately whereas Clustering is done based on the characteristics e.g., 1) Animals with 2 legs 2) Animals with 4 legs.

• Data Classification uses labeled data whereas Data Clustering uses unlabeled data.
• The most popular Data Classification algorithms in data mining are the K-Nearest Neighbor and decision tree algorithms.
• The two common Data Clustering algorithms in data mining are K-means clustering and hierarchical clustering.
• Data Classification output is known.
• Data Clustering out is unknown.
• Training data is provided for Data Classification.
• No training data is provided for Data Clustering.

This is a very confusing topic as both Categorization and Classification are used interchangeably.

Both Categorization and Classification are Metadata and are used for Data Governance as well.

Categorization is based on the attributes and properties of entities in dataset whereas Classification is based on the business requirements.

As you can refer to the image, data has been categorized in passport, health record, financial record, job posting etc., and at the same time these categories are classified as Medium Risk.

‘Both these terms fall under Supervised Learning in Data Science domain.’

For example, there are grandparents, parent, children, cousins, servants, gardener, machinic, car painter etc., all these falls under categories but there can be further classifications i.e., family members and non-family members.

Categories: Grandparents, Parent, Children, Cousins, Gardener, Machinic, Car painters

Data Annotation and Data Labelling works together to prepare dataset for Machine Learning Models in Data Science practice.

‘Data Annotation is the technique of doing Data Labelling.’

Data Annotation is a complete process with many steps to curate, whereas Data Labelling is ONE of the many steps in it.

Data Labelling is about tagging the data for training e.g., machine learning models whereas Data Annotation is the technique for tagging the data so ML models can reuse it as well. Data Labelling tags are used by ML models for future pattern recognitions whereas Data Annotation is about recognizing the data via a process.

Just like Data Labelling, Data Annotation is also for supervised machine learning to prepare data as an input and to store it for future reference. Not to mention, there are STILL many organizations doing manual Data Labelling. Data Annotation is about the process so Data Labelling can be automated for accuracy, eliminating human intervention. Bad Data Labelling can impact the outcome of applications.

Data discovery involves the collection and evaluation of data from various sources and is often used to understand trends and patterns in the data. But understanding trends and patterns cannot be simply done based on data coming from sources such as.

• Data can have Null values.
• Data can have different types of date formats.
• Data can have spelling mistakes.
• Data can have special characters.
• Data can have different codes for the same product e.g., a Glass code can be GLS in one source system and GS in another. So, it's very important to streamline the data sets before presenting them to end users.
• Data can have duplicates.
• And many more....

Data Discovery is different for different roles i.e., for Data Scientist or Data Analyst or Business End User or for C-Level.

• Data Scientists mostly prefer raw data without any data cleaning, cleansing, and scrubbing for Data Dredging, Snooping, p-hacking, and Fishing [there are separate topics on these terms].
• Data analysts mostly prefer data after it’s at least cleaned and cleansed.
• Business End Users and C-Level expect data to be cleaned, cleansed, scrubbed, wrangled, and munged [there are separate topics on these terms] before it reaches them for decision making. Well, a fair expectation.

“Semantic” is an ancient Greek word: σημαντικός sēmantikós, "significant" is the study of reference, meaning, or truth.

In Data World, Semantic layer is built for the sole purpose of supporting business users.

Semantic layer is to convert complex data models into simple and user-friendly interface for business users.

A semantic layer is the representation layer on top of databases, schema(s), tables and columns in such a way that Business users who have no technical know-how should be able to drag and drop required details on Dashboards. For drag and drop, first, one must assign business names to all databases, schema(s), tables and columns before exposing in BI tools. For example, there can be a column name as Date but the data in this column can be Date_Of_Birth or Transaction Date or Financial_Year_End_Data. Converting data models, column/ table/ schema/ database names as per business understanding is semantic process.