Knowledge Hub

Data Harmonization is normally confused with Data Integration, Data Blending, Data Warehousing, Data Marts etc. Yes, all do the same steps of connecting and bring all required data from all the disparate sources in one central location, but Data Harmonization goes one step ahead.

Data Harmonization standardizes data coming from multiple source systems in one unified format, so it becomes a bridge between data and business requirement. When we say, one unified format, it means data is converted in such a way that it can become accurate for someone.

Data Harmonization can be of many types.

• Harmonizing data in one location for a specific department can be called a Data Mart.
• Defining standard naming conventions, data type, codes coming from multiple sources in one unique format can be Harmonizing data for Master Data Management.
• Gathering Metadata details of all source systems, data lakes, data warehouses, data marts up to reports and dashboards is Harmonizing data for Data Lineage.

Point-to-Point model (PP) connects all applications with each other e.g., if there are 5 applications then there will be many connections as depicted in below picture.

The Hub-and-Spoke model (SP) creates a new centralized connection points called as Hub and all applications will ONLY connect with centralized ONE Hub.

We can clearly see Hub-and-Spoke model is much better in access management, flexible, cheaper, secure etc. as compared to Point-to-Point model.

Examples of Hub-and-Spoke model are Data Warehouse (dimensional), Data Hub, Data Fabric etc. which serves as single point of truth for all enterprise users.

Key Items

• User Access: SP model can management user access matrix at centralized place as compared to PP model.
• Security: As all users will be connecting to one centralized Hub, risk of security breach will be less in SP model as compared to PP model
• Flexibility: In SP model, Hub is easy to incorporate those new requirements which doesn’t need to implement in source systems.
• Cheaper: In SP model, though Hub implementation requires initial investment but once it’s implemented all new requirements cost will have much cheaper impact as all changes will go in Hub as compared to each application in PP model.

Syndication is also known as synchronization, yes now it must be making sense to you 😊.

‘Data Syndication is a process not a technology.’

The word syndicate comes from the French word syndicate which means "trade union" (syndic meaning "administrator"), from the Latin word syndicus which in turn comes from the Greek word σύνδικος (syndikos), which means "caretaker of an issue"; compare to ombudsman or representative.

‘Data Syndication is to collect and prepare update-to-date dataset for a particular activity.’

For example, prepare a dataset from multiple source systems for the marketing team to plan their upcoming campaigning exercise.

Data Syndication process

• Can handle any type of format as input and output as well.
• It can send alerts.
• It can configure access matrix.
• It provides monitoring dashboards and reports for compliance and audit trials.
• As it’s a process, any expensive or cheap technology can be utilized.
• Output formats can be designed based on customer requirements.
• Can send email, SMS, spreadsheet, database, ML models etc.

Data Supply Chain is not a data capability rather a process.

Just like generic supply chain, Data Supply Chain works in the same manner where it starts from Source Data Stage 🡪 Transformation and Integration Stage 🡪 Consumption Stage

• Source Data Stage is the first stage where raw data is planned and identified i.e., OLTP systems.
• Transformation and Integration Stage is where data is converted based on business requirement in-shape of Data Lake Systems or Data Warehouses
• Consumption Stage is where data is store for business use in-shape of Data Lake Systems or Data Warehouses (OLAP systems) or Semantic Layer etc.

Data Supply Chain is a very critical operational process to manage Data Asset and Liabilities. In the absence of visibility on Data Supply Chain, organization cannot plan ROI as well as TTM which is critical for the Data Monetization.

Note: Data Assets, Data Liabilities and Data Monetization all are explained in separate topics.

Data Fusion, another term overlapping with many other already in-place terms.

‘Data Fusion is about pulling same nature of data from multiple source systems to co-relate with each other to produce one unified output.’

For example, data for a particular product can come from POS (point of sales), from shops, from online stores etc. At the same time, people can talk about that product on social media, in seminars, webinars etc. Fusion of all those data elements can produce valuable information about that product for sales and marketing teams.

Yes, looking into image you can make out same thing can be achieved in Data Warehouse, Data Mart, Data Lake, Data Lakehouse, Data Vault/ Business Vault etc., and you are not wrong. All above-mentioned Data Store can be used for Data Fusion.

Integration Hub is normally confused with datastores like Data Warehouse, Data Lake, Data Lakehouse etc., which is not correction.

Integration Hub is a single central place from where you can send instructions to internal and external systems to perform some actions. Integration Hub consists of Spokes, each Spoke consists of number of steps for actions.

Generics Steps

• Create actions to be sent out to external systems.
• Run commands on external servers.
• Allow sending messages e.g., Rest, SOAP etc.
• Allow parsers.
• Allow SQL command to databases.

Yes, we do it every day e.g., we create connections with any system e.g., creating a connection with SharePoint then running commands to Delete a file. Consider 100 users want to do the same action, each must go through the same number of steps. Using Integration Hub, Spokes consisting of connections, credentials, and actions to be performed on e.g., SharePoint. Now, all those 100 users can reuse the same Spoke created to delete files in SharePoint.

A Data Abstraction Layer (DAL) is like a special helper in computer science and software engineering. It's a layer of code that keeps the tricky details of how data is stored or used, separate from the rest of the software.

Here are some important things to know about a Data Abstraction Layer:

• Abstraction: It means hiding all the complicated stuff about storing data behind a simpler way of working with it. This makes it easier for developers to handle data without worrying too much about the nitty-gritty details.
• Separation of Concerns: The Data Abstraction Layer helps keep things organized by separating how data is stored from how the software works. This makes the software easier to understand and take care of.
• Consistent Interface: The layer provides a set of clear rules that the rest of the software follows to work with the data. This is helpful because it keeps the software running smoothly even if the way data is stored changes.
• Adaptability: The Data Abstraction Layer acts like a shield, protecting the software from changes in how data is stored. So, if the way data is kept in the computer changes, only the Data Abstraction Layer needs to be adjusted, and the rest of the software can stay the same.
• Security and Validation: The Data Abstraction Layer also looks out for the security of the data. It makes sure that data is handled in a safe and controlled way.

Data Interchange refers to the electronic exchange of information or data between different computer systems, applications, or software. It involves the transmission and reception of structured data to enable seamless communication and integration between disparate systems.

Few Key Points:

• Structured Data Exchange: Data interchange involves the exchange of structured data, often in a standardized format that both the sending and receiving systems can understand.
• Cross-Platform Communication: It facilitates communication between diverse platforms, applications, or software that may have different data structures or operate in distinct environments.
• Standards and Formats: To ensure interoperability, data interchange often relies on industry standards and common formats. Examples include XML (eXtensible Markup Language), JSON (JavaScript Object Notation), or specific data exchange standards in various industries.
• Facilitates Integration: Data Interchange plays a crucial role in integrating different systems and enabling them to work together cohesively. It allows for the sharing of information between applications, databases, or even organizations.

Data Replication is the process of creating and maintaining multiple copies of the same dataset in different locations or systems. The goal is to ensure data availability, improve fault tolerance, and enhance performance by distributing data across various storage locations. Replicated data can be synchronized in real-time or at scheduled intervals to keep the copies consistent.

For example, imagine you and your classmates are working on a group project, and you need to share a document with everyone. Instead of having a single copy stored on one person's computer, you decide to use data replication to ensure that everyone has access to the most up-to-date version.

Here's how it works:

• Initial Copy: You create a document on your computer with the project details. This is the initial copy.
• Replication: You decide to replicate the document by making copies and distributing them to each group member. Now, everyone has their own copy of the document.
• Changes and Updates: As the project progresses, different group members make changes to their copies based on their contributions. For example, one person adds information about research findings, another updates the formatting, and so on.
• Synchronization: Periodically, or when major changes occur, you decide to synchronize the copies. This means that everyone shares their updates, and each person gets the latest version of the document. This ensures that all copies are consistent and reflect the most recent changes.

Data Collaboration in the data world refers to the practice of sharing and working with data across individuals, teams, departments, organizations, or even between different systems to achieve common goals, solve problems, or generate insights. It involves cooperative efforts in handling, analyzing, and deriving value from data assets. Data Collaboration is essential for fostering innovation, improving decision-making, and addressing complex challenges that may require diverse perspectives and expertise.

Key aspects of Data Collaboration include:

• Sharing Data: Organizations and individuals share data sets, information, or insights with one another. This can include internal data sharing within an organization or external sharing between different entities.
• Interdisciplinary Cooperation: Data Collaboration often involves collaboration between individuals with diverse skills and expertise. For example, data scientists, analysts, domain experts, and business professionals may collaborate to extract meaningful insights from data.
• Data Integration: Integrating data from various sources to create a unified view. This could involve merging datasets, connecting databases, or using middleware to facilitate seamless data exchange.
• Secure Data Exchange: Ensuring that data sharing is done securely and in compliance with privacy regulations. This involves implementing measures to protect sensitive information during the collaboration process.
• Open Data Initiatives: Participation in open data initiatives where organizations contribute and access shared datasets for the benefit of the wider community or industry.
• Cross-Functional Projects: Collaborative projects that span different functions or departments within an organization. For example, marketing, finance, and operations teams working together on a data-driven project.
• Data Standards and Governance: Establishing data standards and governance practices to ensure consistency, quality, and reliability of shared data. This helps in maintaining trust among collaborators.