Knowledge Hub

If you would like to step into the data domain, the most significant topics anybody must recognize are Metadata, Master & Reference Data Management. If you grasp these areas, one will require 80% less time to orchestrate any Data Solution.

Master Data refers to the core datasets that are essential for business operations and act as the foundation for all other data within an organization. These datasets typically represent key entities or subjects that are critical to the functioning of the business and are used across multiple systems, processes, or applications.

A unique characteristic of Master Data is that it cannot exist independently, it relies on the existence of specific entities or attributes to be meaningful. For example, you cannot have a customer master record without defining key attributes such as the customer’s name, address, and unique identifier (e.g., customer ID). Similarly, a product master record cannot be created without information such as the product name, SKU, or category.

Data Deduplication is mostly involved in ETL or ELT process.

‘Data Deduplication is about removing duplication in the data.’

There are two kinds of data which can be duplicated i.e., 1) Raw or Processed Data and 2) Absence of Golden Record

Types

1. Raw or Processed Data: With-in this type of data, there are many ways to have duplicate data but below are two main reasons why mostly it ended up generating duplicated data.
   1. Data entered from front-end applications like when system is so badly designed that it allows to enter duplicate data e.g., a customer is wrongly created twice, or a product sold once but system allows to enter it more than one time as human error. This data must be removed as part of the Data Quality process.
   2. Data transformed during ETL/ ELT process from source systems to Data Warehouse or Data Marts or Semantic Layers etc. This data also needs to be removed by Data Quality process.
2. Absence of Golden Record: Duplicates also occur when golden records are missing? What is Golden Record? Absence of Golden Records means, when there is more than one code for single entity e.g., consider an insurance company has a SAME TYPE OF POLICY with policy code as 001 in one system and 002 in second system and 003 in third system, now when data from all these three systems flows into a centralized location which can be a Data Warehouse or a Data Mart there will be transactions related to each of these policies but in reality all belong to SAME TYPE OF POLICY. When this data goes to the marketing department and they plan to send online campaigns, instead of sending 1 campaign for 1 type of policy, there will 3 campaigns designed for the same 1 type of policy. This scenario can be handled by a de-duplicated Golden Record with the help of Master Data Management (MDM) Application.

Data Hierarchy is a known term and most of us are aware of it but sometimes, it’s confused with Data Taxonomy & Data Ontology.

Data Taxonomy and Data Ontology are metadata for the purpose of Data Governance whereas Data Hierarchies presents actual data. A very well-known and famous example is the Organizational Chart where hierarchy starts from CEO and goes down to Database Administrator in data domain.

• ‘Data Taxonomies apply to identified objects or phenomena.’’
• ‘Data Ontologies apply to terms, concepts, or categories.’
• ‘Data Hierarchy is a chain of relationship in actually data.’

Family tree is another simple example.

Data can be grouped in a way where data exploration and representation become easy. For example, at database level, hierarchy can be Database 🡪 Files/ Tables 🡪 Records 🡪 Fields/ Columns 🡪 Bytes 🡪 Bits.

Same line Data Taxonomy, Data Hierarchy also have single parent and multiple child relationship among the rows of dataset.

Syndication is also known as synchronization, yes now it must be making sense to you 😊.

‘Data Syndication is a process not a technology.’

The word syndicate comes from the French word syndicate which means "trade union" (syndic meaning "administrator"), from the Latin word syndicus which in turn comes from the Greek word σύνδικος (syndikos), which means "caretaker of an issue"; compare to ombudsman or representative.

‘Data Syndication is to collect and prepare update-to-date dataset for a particular activity.’

For example, prepare a dataset from multiple source systems for the marketing team to plan their upcoming campaigning exercise.

Data Syndication process

• Can handle any type of format as input and output as well.
• It can send alerts.
• It can configure access matrix.
• It provides monitoring dashboards and reports for compliance and audit trials.
• As it’s a process, any expensive or cheap technology can be utilized.
• Output formats can be designed based on customer requirements.
• Can send email, SMS, spreadsheet, database, ML models etc.

Data Standardization can be imposed at a data level or at a data store level.

Data Standardization means converting data into such a format that it’s accessible, useable, readable, and actionable.

Data Standardization reduces manual efforts, increase efficiency, enable segmentations, remove duplications, increase data quality, enforce data governance, increase business opportunities, improve analysis and analytics.

Steps for Data Standardization

• Know your data and what it is required for.
• Recognize data entry points.
• Outline the data standards.
• Standardize your data with the help of data scoring.

Data Standardization Approaches

• Converting all dates format into single format.
• Arranging or sorting database on requirements.
• Converting unstructured data set into structured data set e.g., video, audio, image, pdf, email, word etc. in one readable format.
• Converting multiple semi-structured formats in one format e.g., JSON and XML in one readable format.
• Having Gold Records using MDM system.
• Via Data Profiling process
• Via Data Transformation process
• Via Data Virtualization environment
• Data pipeline processing at one given time i.e., real-time, daily, monthly etc.
• Exposing data in the shape of Reports and Dashboards.
• Converting all data from different language to one language e.g., English.
• Bring data from all relevant source systems into one Unified Data Model in a Data Warehouse or in Star or Snowflake Schema or in Semantic Layer or having a multi-dimensional Cube.
• Enrichment of metadata of the data and Creation "Quality gates" that indicate freshness and anomaly.

Geocoding has become of the most demanding data element in current era where data is generated everywhere and by everyone.

‘Geocoding refers to the geographic location.’

Geocoding is about the name, street, city, regions etc., rather exact location on Earth. The exact data elements representing Geocoding are Longitude and Latitude, also known as Lat and Long. Using Lat and Long, one can reach the exact location, anywhere in the World.

In the current era social media, mobile applications, sensors, satellite scans, CCTV, online video(s), audio(s) etc., all have become a norm. Millions of users are using mobile on a daily basis, everywhere, all the time around the globe. Location information has become one of the most important pieces of information for everyone especially consumer details.

For example, we all know Amazon, it’s one the most used online stores around the globe which is used by millions of people to buy and sell products, 24/7. If Amazon does not know where the buyer is from, and what is the location where most buyers of a particular product are, then it can’t plan its marketing campaigns.

Data Matching and Linking is for multiple purposes across systems.

Data Matching and Linking is about matching and linking datasets across the systems to find out which is the best record to link with. Generating Golden Record for MDM (explained separately) can be one of the use cases. Additionally, we are also referring to when the same nature of data is coming from multiple source systems and there is a need to find the best possible record.

Please note, we are not talking about manual matching and linking, we are talking about complex software(s) using complex algorithms to find the right data.

For example, customer onboard process can be done Online or Customer Service Centre or at a Branch. If the same customer gets registered 2 to 3 times via different channels, then one must use Data Matching and Linking process to identify customer with high weightage.

Types

• Referential is where exact fields are matched and linked like National ID, Social Security Number, Password numbers etc.
• Probabilistic is where multiple fields are used like name, date of birth, parent’s names etc.
• ML Association is where machine learning searches for matching data element to link.

‘Disambiguation means to make something clear and understandable.’

We all know the word i.e., Ambiguous which means something which is unclear especially when same words are used for multiple purposes.

In the current era, semi and unstructured data is the norm. Now a days, social media has become one of the major sources for customer profiling. Then audio and videos are another source which can have any kind of data behavior.

‘Data Disambiguation’s biggest use case is to handle Sarcasms.’

As we know, in Sarcasm, dual meaning words are heavily used. For example, there is a common saying i.e., What’s up? This phrase can be used to say greeting to a friend and can also be used to show anger to someone unknown moving around in your restricted area.

The job for Data Disambiguation process is.

• To dig out the right meaning out of confusing statements.
• To understand the right concept of statement.
• Remove anything which can corrupt the behavior of pattern.
• Translate or replace dual meaning word with single meaning words so end user is not confused before taking any decision.

ISO 11179 is an international standard that provides guidelines for the representation and management of metadata for describing data and information resources. In simpler terms, it offers a standardized way to organize and describe data, making it easier to understand, share, and manage.

Let's break down ISO 11179 in more detail:

As we have read in Metadata Management topic, Metadata is Data about Data. It describes the characteristics, properties, and context of data, helping users understand what the data is, how it was collected, and how it can be used.

ISO 11179 focuses on standardizing the way metadata is defined, organized, and managed. It provides a common framework for describing data elements, ensuring consistency and clarity in data descriptions. The standard helps organizations create a common language and structure for describing data, making it easier for different systems and people to understand and use the data.

ISO 3166, one type of Reference Data, is an international standard that defines codes for the names of countries, territories, and special areas of geographical interest. In the data world, ISO 3166 codes are widely used for various purposes, such as uniquely identifying countries in databases, international transactions, and data exchange.

Let's break down ISO 3166 in more detail:

Country Codes: ISO 3166 provides three types of codes for each country or territory:

• ISO 3166-1 Alpha-2: Two-letter country codes (e.g., US for the United States, CA for Canada).
• ISO 3166-1 Alpha-3: Three-letter country codes (e.g., USA for the United States, CAN for Canada).
• ISO 3166-1 Numeric: Three-digit numeric codes.

ISO 3166 standardizes the representation of country names to avoid confusion in international communication, commerce, and data exchange. The standard is maintained by the International Organization for Standardization (ISO) and is periodically updated to reflect changes in country names and geopolitical developments.