Knowledge Hub

Data Inventory is a very simple term but sometime get a little bit confused with Data Catalog and Data Assets etc.

Data Assets is about knowing which kind of data we have in our organization with monetary value.

Data Catalog is about knowing system details, databases, tables, and columns and then mapping those with business requirements.

Data Inventory is about consolidating information from Data Asset and Data Catalog with addition details like.

• where this data comes from
• who generated this data?
• when it was generated
• frequency of inserts, updates, and deletes.
• contents details
• security and privacy approaches
• metadata management approaches
• data quality standards
• transformation details
• business rules
• data models
• list of software
• name of all the applications
• licensing details
• data sources details
• target systems details.
• owners of source and target systems

We all maintain inventory of something or the other, either it’s at home or office. A simple example is Passwords Inventory, now a days there are many social media applications then Hotmail, Gmail, etc. We normally maintain all those passwords somewhere safe right so that is your Password Inventory.

Data Curation is about generating, managing, and maintaining data for future use.

‘Curation comes from the Latin word curae, meaning "care for" -- a meaning it still has today, including in relation to data.’

Data Curation is about taking care of the data from start till end. From extracting till loading. From management to governance. From raw till transformed. From generation till secured and private storage. From open till encryption. As the name says, CARE FOR. Data Curation is also about taking steps to preserve data for future reference as well. Data Curation is the process of preparing data for business and for decision support systems. There is nothing complex about its definition 😊.

Steps for Data Curation

• Understanding the data
• Transformation
• Metadata Management
• Cleansing
• Reconciliation

Stages of Data Curation

• Storing --> Sharing --> Discoverable

Data Contextualization has emerged as a critical factor in dealing with and summarizing vast, rapidly changing, and diverse unstructured datasets. Let's break it down.

We all understand the concept of context. For instance, when you're about to commence a meeting with management, you begin by acknowledging them and stating the purpose of the meeting. This establishes the context, ensuring that the audience knows what to expect and how to interpret the upcoming presentation.

Similarly, in the realm of data, Data Contextualization has become a fundamental process, especially when extracting insights from unstructured data. Unstructured data typically contains a substantial amount of information that is irrelevant to decision-making. The relevance of a data element may vary depending on the specific decision being made.

Here's why Data Contextualization is essential:

• Filtering Irrelevant Data: It eliminates irrelevant data, reducing the overall volume of data. This, in turn, simplifies data processing, cleaning, transformation, and maintenance.
• Enhancing Decision-Making: By contextualizing data, you can combine external data with internal data, providing a more comprehensive foundation for decision-making.
• Providing Clarity: It offers a clear rationale for the data, making it easier to comprehend and analyze.
• Adding Points of Interest: Data Contextualization allows for the inclusion of relevant insights or points of interest about the data, facilitating better understanding.

Semantics are different for everyone. In an organization, each department has its own semantics, to view data. Same data can have different meanings for one department as compared to others.

Salary for HR department is to know who is getting what and when they hire new candidate, they can refer to what others are getting based on role, JD, skill set etc.. But Salary for Finance department can be, to calculate annual expense for the company.

Semantic Layer in Decision Support Systems are based on business requirements, in their language, using their own key words, their codes etc.

Interoperability on the other hand is about exchanging something across systems so others don’t need to regenerate it again and again.

Semantic Interoperability is about exchanging semantic information across systems or across departments or across organizations. But this comes with a challenge whether the receiving party can understand the keywords, symbols, terminologies etc., from the sending party.

ISO 11179 is an international standard that provides guidelines for the representation and management of metadata for describing data and information resources. In simpler terms, it offers a standardized way to organize and describe data, making it easier to understand, share, and manage.

Let's break down ISO 11179 in more detail:

As we have read in Metadata Management topic, Metadata is Data about Data. It describes the characteristics, properties, and context of data, helping users understand what the data is, how it was collected, and how it can be used.

ISO 11179 focuses on standardizing the way metadata is defined, organized, and managed. It provides a common framework for describing data elements, ensuring consistency and clarity in data descriptions. The standard helps organizations create a common language and structure for describing data, making it easier for different systems and people to understand and use the data.

A MetaGraph provides information about the structure, schema, and semantics of a knowledge graph within a graph database. It essentially acts as a "graph of metadata" that defines how entities (nodes) and their relationships (edges) are organized and interact, including their attributes, constraints, and interdependencies.

‘The Metadata of Knowledge Graphs in Graph Databases.’

In the world of data management, the term MetaGraph represents an advanced layer of abstraction that encompasses the metadata of a knowledge graph. While knowledge graphs are designed to represent relationships between entities, a MetaGraph serves as the blueprint that defines the structure, schema, and semantics of those relationships. This topic delves into the concept of MetaGraph, their importance, use cases, and their impact on the evolving landscape of graph databases.

MetaGraph is an enabler for a complete digital twin of simple to complex reality. The MetaGraph technology offers a cutting-edge approach to define, manage and utilizing metadata in a digital ecosystem – thus providing rich context to data across functions and systems. This is critical for AI, machine learning and data driven as quality in decisions based on facts become more central to industry transformations. The demand for high-quality, relevant data has never been greater. In the end the MetaGraph can enhance how businesses operate and innovate.

DOGs, or Data Object Graphs, represent data objects and the relationships between them in the form of a graph. Instead of organizing data only in tables and rows, this approach visualizes data as connected objects.

In a Data Object Graph:

• A node represents a data object (such as Customer, Product, Order, or Employee).
• An edge represents a relationship between those objects (such as buys, works in, manages, owns).

This way of representing data focuses on how objects are connected rather than how they are stored.

Why Data Object Graphs Matter

Traditional relational systems organize data into tables. While this structure is powerful, it sometimes hides the complexity of relationships. In real-world business environments, data is deeply interconnected.

For example:

• A customer places multiple orders.
• An order contains multiple products.
• A product belongs to a category.
• An employee manages multiple customers.

In simple words, Graph Theory defines the structure i.e., data model, MetaGraph are the metadata, Taxonomies are the classifications, and Ontologies are the rules.

MetaGraph, Taxonomies, and Ontologies are essential tools in knowledge management, data governance, and AI, each serving distinct yet occasionally overlapping purposes. Below is a detailed breakdown of their characteristics and applications.

MetaGraph

MetaGraph represents metadata within a graph database or knowledge graph, providing a high-level overview of the data's structure, relationships, and properties.

Key Characteristics:

• Focus: Outlines the data model, specifying entities (nodes), relationships (edges), and their attributes.
• Structure: Functions like a schema, defining node and edge types, along with constraints.

Purpose:

• Aids in understanding the graph's structure.
• Supports data governance, lineage tracking, and semantic clarity.
• Enables enhanced searchability and automation by contextualizing graph content.

Data Wrapping is the practice of augmenting raw data with additional layers of value, such as tools, metadata, and context, to make it easier to use, understand, and act upon. This process is crucial in transforming data from a passive resource into an actionable asset.

The Core Elements of Data Wrapping

Data wrapping goes beyond merely presenting raw data, it involves transforming it into a meaningful and actionable resource. Here are the core elements, expanded with practical insights:

1. Metadata and Context

Metadata acts as the "data about data," providing essential context that helps users interpret and utilize data effectively.

• What It Includes: Descriptions of data sources, timestamps, definitions, units of measurement, and relationships between datasets.
• Why It’s Important: Without metadata, users may misinterpret data or struggle to understand its significance.
• Example: A dataset tracking customer orders might include metadata about the region, the time zone in which orders were placed, and the categorization of product types.