Knowledge Hub

Consumption layer is confused with Serving Layer due to its name. Folks relates it that the layer which is used by Consumers to extract data. The Layer which Consumers use to extract information is Serving layer (explained separately).

Consumption Layer, as its name says it all, is a layer which consumes or takes-in data from source systems. In the current era, due to huge amounts of data coming in every second from sources like social media, CCTV, images, audios, sensors data etc. technology was required to consume data on a fast pace in MPP mode.

Tools like Hadoop, Object Storage and many RDBMS are giving MPP features now a days.

In Lambda Architecture, Serving Layer is the last layer, Consumption Layer is the layer which process data from source till Serving layer.

In the image, Batch and Speed Layers are Consumption Layers. In an OLAP system, the consumption layer is Data Warehouse and Data Mart, considering there are OLAP Cubes and Semantic layers but in a Data Warehouse, consumption layer is Landing and Staging zone.

We all have heard about MPP (Massive Parallel Processing). In the current era, consumption and serving layer without MPP capability cannot survive.

MPP is mostly used to read and write (mostly insert) in parallel where more than 1 processor is involved.

In MPP, multiple processers are used to process queries in parallel which might be running in single node or across multiple nodes, in same location or multiple locations.

In a non-MPP mode when a query is executed its different pieces of code are executed one by one in a sequence. But in MPP mode when a query is executed its different pieces of code are executed in parallel on all the processors in one go. This increases the execution of queries 100 times faster than compared to one-by-one execution.

In the current era where data coming from social-media, audios, videos etc., are coming in such a velocity/ pace/ speed that without MPP, it will not be possible to write, process or read data in timely manner.

Coupling is a parameter dependency approach used cross software or cross systems.

This approach is used when there are more than one software or system is in the picture to deliver the result.

Data Coupling is about when data elements are used, to kick-off the next step in a different system. For example, in data pipeline, we are pulling data from a hospital management system e.g., when a baby is born let’s have a data dependency like, the moment the baby is registered in system, pull its data. So, it’s a cross-system data dependency or we called Data Coupling.

The disadvantage of Data Coupling approach is, if something goes wrong in the first system, dependent systems cannot move.

Let me share another old example. Old data pipeline folks will remember this one i.e., when we used to load files from another system, we had to wait till the time whole file movement is completed. The challenge was how to make sure when to start pulling like only once file is completely copied.

Integration Hub is normally confused with datastores like Data Warehouse, Data Lake, Data Lakehouse etc., which is not correction.

Integration Hub is a single central place from where you can send instructions to internal and external systems to perform some actions. Integration Hub consists of Spokes, each Spoke consists of number of steps for actions.

Generics Steps

• Create actions to be sent out to external systems.
• Run commands on external servers.
• Allow sending messages e.g., Rest, SOAP etc.
• Allow parsers.
• Allow SQL command to databases.

Yes, we do it every day e.g., we create connections with any system e.g., creating a connection with SharePoint then running commands to Delete a file. Consider 100 users want to do the same action, each must go through the same number of steps. Using Integration Hub, Spokes consisting of connections, credentials, and actions to be performed on e.g., SharePoint. Now, all those 100 users can reuse the same Spoke created to delete files in SharePoint.

A Data Abstraction Layer (DAL) is like a special helper in computer science and software engineering. It's a layer of code that keeps the tricky details of how data is stored or used, separate from the rest of the software.

Here are some important things to know about a Data Abstraction Layer:

• Abstraction: It means hiding all the complicated stuff about storing data behind a simpler way of working with it. This makes it easier for developers to handle data without worrying too much about the nitty-gritty details.
• Separation of Concerns: The Data Abstraction Layer helps keep things organized by separating how data is stored from how the software works. This makes the software easier to understand and take care of.
• Consistent Interface: The layer provides a set of clear rules that the rest of the software follows to work with the data. This is helpful because it keeps the software running smoothly even if the way data is stored changes.
• Adaptability: The Data Abstraction Layer acts like a shield, protecting the software from changes in how data is stored. So, if the way data is kept in the computer changes, only the Data Abstraction Layer needs to be adjusted, and the rest of the software can stay the same.
• Security and Validation: The Data Abstraction Layer also looks out for the security of the data. It makes sure that data is handled in a safe and controlled way.

Technical Debt, often abbreviated as "tech debt," is a metaphorical concept in software development that refers to the implied cost of additional work created when software is built or modified in a way that prioritizes short-term goals or speedy delivery over long-term sustainability. It is similar to financial debt, where you borrow money with the understanding that you will pay it back with interest.

Technical debt arises when developers choose a quick and easy solution that may solve an immediate problem but is not necessarily the best or most efficient in the long run. This can result from factors such as tight deadlines, evolving requirements, or a need for rapid prototyping.

There are several types of technical debt, including:

Code Debt: Poorly written or poorly structured code that might work but is difficult to understand, maintain, or extend.

Design Debt: Rushed or inadequate system or software architecture that may lead to inefficiencies, difficulties in adding new features, or challenges in maintaining the system.

Data Warehouse

A data warehouse is a type of data platform that is designed to store and analyze structured data from various sources, providing a central repository for business intelligence (BI) and analytics applications. The data stored in a data warehouse is typically historical, integrated, and cleansed, and can be used to support decision-making, reporting, and analysis at all levels of an organization. The data stored in a data warehouse is typically structured and organized into tables and is optimized for efficient querying and analysis.

Data Lake

A data lake is a centralized repository that allows you to store all your structured, semi and unstructured data at any scale. The term was coined to represent the idea of a system where data flows in from various sources and resides in its most "natural" state, just like water in a lake. These data can range from raw, unprocessed files to transformed, structured records.

Reference Architecture is foundational to designing any complex system or solution. It provides a high-level diagram that represents the structural pattern of the solution, establishing a consistent framework that guides the design and implementation of similar projects across an organization. Unlike solution architecture, which is customized for each project, reference architecture offers a generalized template that aligns with established best practices, making it broadly applicable and easily understandable for both technical and non-technical stakeholders.

Purpose and Benefits

The main objective of a reference architecture is to set a standardized approach for designing solutions within an organization. This ensures consistency, reduces redundancy, and accelerates solution delivery by offering a clear blueprint. With this architecture, teams can quickly understand the fundamental elements of the solution, enabling efficient alignment with organizational goals. Additionally, reference architectures aid in risk management by promoting tried-and-tested design patterns and technologies, minimizing potential errors in execution.

Before we explore what constitutes a Data Product (DP), it is important to establish the foundational context within which these products exist. Most enterprise data environments comprise two distinct but interconnected types of systems:

• OLTP (Online Transaction Processing) systems, which are responsible for generating operational data through transactional processes.
• OLAP (Online Analytical Processing) systems, which are designed to serve data for analytical and decision-support purposes.

We will not dive into the technical specifics of OLTP and OLAP here, as those are covered in dedicated sections elsewhere in this book. However, it is essential to understand this distinction because Data Products are primarily built and operated within the OLAP ecosystem.

In essence, OLTP generates data. OLAP serves it.

Data products are almost always born, shaped, and consumed in the OLAP layer, where raw data is transformed into meaningful, trustworthy outputs for business consumption.

In the modern data landscape, few topics spark more debate than the question: “What is a Data Product?”

Across conferences, boardrooms, and industry discussions, the term has taken on a life of its own. Many practitioners argue over definitions, frameworks, and taxonomies, yet the underlying truth is simpler: most of us already know what a data product is.

The real challenge isn’t defining it. It’s understanding its quality.

At its core, a data product is any output, a curated dataset, analytical model, dashboard, that delivers tangible business value, offsets the cost of inaction (COI), and can be reused by multiple stakeholders. By that definition, its importance is undeniable.

But the real question is: Is the data product Good, Bad, or Ugly?

The Juice Analogy

To make this concept tangible, think about juice as an analogy.

When someone makes juice for you at home, is it a product? No. Even if it has the same ingredients, taste, and nutrition as a bottled juice, it’s not a product because: