Knowledge Hub

Consumption layer is confused with Serving Layer due to its name. Folks relates it that the layer which is used by Consumers to extract data. The Layer which Consumers use to extract information is Serving layer (explained separately).

Consumption Layer, as its name says it all, is a layer which consumes or takes-in data from source systems. In the current era, due to huge amounts of data coming in every second from sources like social media, CCTV, images, audios, sensors data etc. technology was required to consume data on a fast pace in MPP mode.

Tools like Hadoop, Object Storage and many RDBMS are giving MPP features now a days.

In Lambda Architecture, Serving Layer is the last layer, Consumption Layer is the layer which process data from source till Serving layer.

In the image, Batch and Speed Layers are Consumption Layers. In an OLAP system, the consumption layer is Data Warehouse and Data Mart, considering there are OLAP Cubes and Semantic layers but in a Data Warehouse, consumption layer is Landing and Staging zone.

We all have heard about MPP (Massive Parallel Processing). In the current era, consumption and serving layer without MPP capability cannot survive.

MPP is mostly used to read and write (mostly insert) in parallel where more than 1 processor is involved.

In MPP, multiple processers are used to process queries in parallel which might be running in single node or across multiple nodes, in same location or multiple locations.

In a non-MPP mode when a query is executed its different pieces of code are executed one by one in a sequence. But in MPP mode when a query is executed its different pieces of code are executed in parallel on all the processors in one go. This increases the execution of queries 100 times faster than compared to one-by-one execution.

In the current era where data coming from social-media, audios, videos etc., are coming in such a velocity/ pace/ speed that without MPP, it will not be possible to write, process or read data in timely manner.

Coupling is a parameter dependency approach used cross software or cross systems.

This approach is used when there are more than one software or system is in the picture to deliver the result.

Data Coupling is about when data elements are used, to kick-off the next step in a different system. For example, in data pipeline, we are pulling data from a hospital management system e.g., when a baby is born let’s have a data dependency like, the moment the baby is registered in system, pull its data. So, it’s a cross-system data dependency or we called Data Coupling.

The disadvantage of Data Coupling approach is, if something goes wrong in the first system, dependent systems cannot move.

Let me share another old example. Old data pipeline folks will remember this one i.e., when we used to load files from another system, we had to wait till the time whole file movement is completed. The challenge was how to make sure when to start pulling like only once file is completely copied.

Integration Hub is normally confused with datastores like Data Warehouse, Data Lake, Data Lakehouse etc., which is not correction.

Integration Hub is a single central place from where you can send instructions to internal and external systems to perform some actions. Integration Hub consists of Spokes, each Spoke consists of number of steps for actions.

Generics Steps

• Create actions to be sent out to external systems.
• Run commands on external servers.
• Allow sending messages e.g., Rest, SOAP etc.
• Allow parsers.
• Allow SQL command to databases.

Yes, we do it every day e.g., we create connections with any system e.g., creating a connection with SharePoint then running commands to Delete a file. Consider 100 users want to do the same action, each must go through the same number of steps. Using Integration Hub, Spokes consisting of connections, credentials, and actions to be performed on e.g., SharePoint. Now, all those 100 users can reuse the same Spoke created to delete files in SharePoint.

To make it easy, first must understand Traditional Data Stake (TDS). Now, what is TDS lolz 😊. TDS is your On-premises solution which can include any kind of Databases or maybe Hadoop environment as well. Whereas MDS is your tools and technologies on Cloud.

‘MDS vs TDS is more over a comparison between Cloud vs On-premises infrastructures.’

In TDS/ On-premises, if you need new hardware, there is a complete procurement cycle involved. If you need 2 CPU of processing at nighttime and 4 CPU in daytime but you end up paying 4 CPU for 24 hours as you have already bought the hardware 😊.

In MDS, when you are on Cloud, any resource is few clicks away. If you need 2 CPU as night, you only pay when you are using those and same for daytime CPU requirement.

A Data Abstraction Layer (DAL) is like a special helper in computer science and software engineering. It's a layer of code that keeps the tricky details of how data is stored or used, separate from the rest of the software.

Here are some important things to know about a Data Abstraction Layer:

• Abstraction: It means hiding all the complicated stuff about storing data behind a simpler way of working with it. This makes it easier for developers to handle data without worrying too much about the nitty-gritty details.
• Separation of Concerns: The Data Abstraction Layer helps keep things organized by separating how data is stored from how the software works. This makes the software easier to understand and take care of.
• Consistent Interface: The layer provides a set of clear rules that the rest of the software follows to work with the data. This is helpful because it keeps the software running smoothly even if the way data is stored changes.
• Adaptability: The Data Abstraction Layer acts like a shield, protecting the software from changes in how data is stored. So, if the way data is kept in the computer changes, only the Data Abstraction Layer needs to be adjusted, and the rest of the software can stay the same.
• Security and Validation: The Data Abstraction Layer also looks out for the security of the data. It makes sure that data is handled in a safe and controlled way.

Technical Debt, often abbreviated as "tech debt," is a metaphorical concept in software development that refers to the implied cost of additional work created when software is built or modified in a way that prioritizes short-term goals or speedy delivery over long-term sustainability. It is similar to financial debt, where you borrow money with the understanding that you will pay it back with interest.

Technical debt arises when developers choose a quick and easy solution that may solve an immediate problem but is not necessarily the best or most efficient in the long run. This can result from factors such as tight deadlines, evolving requirements, or a need for rapid prototyping.

There are several types of technical debt, including:

Code Debt: Poorly written or poorly structured code that might work but is difficult to understand, maintain, or extend.

Design Debt: Rushed or inadequate system or software architecture that may lead to inefficiencies, difficulties in adding new features, or challenges in maintaining the system.

Data Warehouse

A data warehouse is a type of data platform that is designed to store and analyze structured data from various sources, providing a central repository for business intelligence (BI) and analytics applications. The data stored in a data warehouse is typically historical, integrated, and cleansed, and can be used to support decision-making, reporting, and analysis at all levels of an organization. The data stored in a data warehouse is typically structured and organized into tables and is optimized for efficient querying and analysis.

Data Lake

A data lake is a centralized repository that allows you to store all your structured, semi and unstructured data at any scale. The term was coined to represent the idea of a system where data flows in from various sources and resides in its most "natural" state, just like water in a lake. These data can range from raw, unprocessed files to transformed, structured records.

To succeed with metadata management, you will need an Operational Reference Model. It is the way to describe the structures and relationships of your business metadata.

The Role and relationship of an Operational Reference Model

Conceptual models, logical models, physical models and canonical models have a specific purpose which is either communication and understanding, or representation of an individual system. Operational Reference Models come into play when organizations require an operational layer which bridges multiple systems and functions and provides a common point of reference or shared language for the enterprise. The Operational Reference Model is similar. to an anti-corruption layer and a semantic layer in its purpose but with an important difference, it's owned and governed by your subject matter experts.

Operational reference model governance

An operational reference model is at its core a set of metadata structured with the help of taxonomies and ontologies. Each part, each domain, is governed by the subject matter experts in the organization in accordance with a federated governance approach.

Reference Architecture is foundational to designing any complex system or solution. It provides a high-level diagram that represents the structural pattern of the solution, establishing a consistent framework that guides the design and implementation of similar projects across an organization. Unlike solution architecture, which is customized for each project, reference architecture offers a generalized template that aligns with established best practices, making it broadly applicable and easily understandable for both technical and non-technical stakeholders.

Purpose and Benefits

The main objective of a reference architecture is to set a standardized approach for designing solutions within an organization. This ensures consistency, reduces redundancy, and accelerates solution delivery by offering a clear blueprint. With this architecture, teams can quickly understand the fundamental elements of the solution, enabling efficient alignment with organizational goals. Additionally, reference architectures aid in risk management by promoting tried-and-tested design patterns and technologies, minimizing potential errors in execution.