Knowledge Hub

Flow of all Data Modelling Concepts & Techniques

Data Modelling is a foundational concept in the field of data management and analytics. It involves creating different representations of data structures to support various business and technical requirements.

As mentioned, there are around 37 data modelling technique which can be categorized in 5 layers, mentioned below.

1. Business & Enterprise Modelling
2. Semantic & Information Modelling
3. Core Data Modelling
4. Specialized Techniques
5. Governance & AI Extensions

Let’s see how all data modelling techniques get arranged in these layers.

Enterprise Data Modelling is the practice of creating a single, high-level view of data for the entire organization. Instead of looking at data system by system or project by project, it looks across all business areas and defines how key data concepts are understood at an enterprise level. The purpose is to make sure the organization has a shared understanding of its most important data.

The main goal of Enterprise Data Modelling is to provide a consistent framework for managing data across the whole enterprise. When this framework exists, different departments and systems can work together more easily. Data becomes easier to integrate, compare, and trust, because everyone is working from the same definitions.

This modelling approach brings together data from different subject areas such as customers, products, finance, and operations into one unified model. Rather than allowing each system to define data in its own way, Enterprise Data Modelling defines common entities and relationships that apply everywhere. Because of this, it acts as a strategic blueprint rather than a detailed design for a single application.

Conceptual and Logical Architecture Modelling describes the overall structure of an enterprise’s systems and how they interact, without focusing on specific technologies or physical implementations. It provides a blueprint that explains which systems exist, what roles they play, and how data and processes flow between them. At this stage, the goal is understanding and alignment, not technical configuration.

The primary goal of this modelling approach is to ensure that systems, data, and processes are well aligned with business needs. By clearly defining how different parts of the architecture fit together, organizations can design solutions that are scalable, well-integrated, and easier to govern over time. This helps prevent fragmented systems and poorly connected solutions that grow in isolation.

Conceptual and Logical Architecture Modelling focuses on components and interactions, rather than code, databases, or vendors. At the conceptual level, it presents a high-level view of the architecture, typically showing major domains such as applications, data, and infrastructure. This view answers questions like which systems exist and what their responsibilities are. At the logical level, the model goes a step further by showing how data flows between systems, how integrations work, and how systems are logically grouped, still without committing to specific technologies.

Medallion Architecture is a new term but again just like many other jargons used by Salespeople, this one is no more than a Buzz word. In other words, we already know this, we are already use it if we have a lakehouse implemented.

People who are working with Databricks and have implemented their Lakehouse probably know what this Medallion Architecture is. Let me try to explain in a few lines for the audience reading this book.

Medallion Architecture is sitting in Delta Lake tables, and it is about processing data in multiple layers i.e., Bronze ⇒ Silver ⇒ Gold layer tables.

• Raw Layer: This layer is normally not under Medallion Architecture as it totally Raw whereas Bronze, Silver and Gold all are in Delta Lake tables. So Raw Layer hosts source data which can’t be pushed to Bronze Layer directly. Most of the organizations prefer to land all source systems data in Raw layer (I also prefer this).
• Bronze: Those source systems which have connectors with Databricks/ Delta Lake can land data in Bronze else Bronze source is most Raw layer. Some data standardization transformation happens here like.

Framework for Big Data Processing

Lambda Architecture is an essential concept in the world of Big Data, gaining attention as technology for handling massive data volumes has advanced. It forms the backbone of many decision-support systems by providing a structured approach to data processing, integrating both batch and real-time data workflows. The Lambda Architecture model offers a comprehensive system for handling data from generation to end-user access, balancing speed and accuracy through batch and real-time layers.

The Foundations of Lambda Architecture

Lambda Architecture is grounded in lambda calculus, a formal system in mathematical logic that supports computation using function abstraction, application, and variable substitution. Lambda calculus is a universal model that can theoretically simulate any Turing machine, making it foundational to computer science and software engineering. In data processing, Lambda Architecture uses this mathematical basis to create a dual-layered system that can handle data asynchronously and in real time.

Kappa Architecture is mostly confused with Lambda Architecture which is fair as both processes Streaming as well as Batch data. Then how both are different?

Due to high demand of Real-Time data, Kappa Architecture has emerged but not for all kinds of data sizes just like Lambda Architecture is.

We have gone through in detail (in separate topic) like what is Lambda Architecture so won’t go in details like how Streaming and Batch works. Let’s understand how Kappa is different than Lambda Architecture.

Kappa Architecture offers ONLY stream processing using Queuing tools like KAFKA for both Streaming and Batch datasets. It doesn’t offer separate tools or technologies to process batches. It says, process both Real-Time Streaming and Batch data using same tool as KAFKA but the biggest differentiator here is, it stores data in a separate data store which might not be suitable to store TBs or PBs of dataset. It offers batch related downstream applications to connect directly to its storage.

Unified Data Architecture is about covering all aspects of data with respect to organizational architectures. There is no pre-defined Unified Data Architecture for everyone.

Let’s see how many Architectures we can have, in an organization with respect to Data.

• Architecture for Source Applications i.e., OLTP
• Architecture for Data Warehouse i.e., OLAP and Cubes
• Architecture for Data Lake, Delta Lake, Data Mesh, Data Vault, Business Vault, Data Hub, Data Fabric
• Architecture for Advance Analytics i.e., for Machine Learning and Deep Learning etc.
• Data Virtualization Architecture

As highlighted above, if not all but there can be many Data Architectures in-place in every organization. In the current era, where data has become the central focus element for everyone, one must design Unified Data Architecture for 30,000 feet view before designing Data Architectures for each component mentioned above.

Just like Enterprise Architecture is about Business Architecture, Application Architecture, Technical Architecture and Data Architecture, all these must follow Enterprise Architecture standards. Same way, all Data Architectures in an organization must follow Unified Data Architecture standards.

In today’s rapidly evolving digital landscape, security threats are more sophisticated and pervasive than ever. Traditional perimeter-based security is no longer sufficient to protect your organization’s most valuable asset – its data. This is where Zero Trust Architecture (ZTA) comes into play.

Data breaches can have catastrophic consequences, both financially and reputationally. Zero Trust Architecture ensures that no entity – human or machine – is trusted by default, minimizing the risk of unauthorized access. With millions of IoT devices and diverse infrastructures spanning on-premises and cloud environments, ZTA offers a robust security framework that addresses these complexities through stringent authentication, authorization, and entitlement mechanisms.

Starting with a zero-trust baseline where no one has access, the architecture gradually opens doors through verified credentials and continuous monitoring. This adaptive approach, combined with data encryption, masking, and anonymization, ensures that your defenses are resilient against emerging threats. Implementing Zero Trust Architecture significantly reduces the risk of data breaches, ensures regulatory compliance, streamlines security operations, and future-proofs your security strategy.

NEON-Inspired Architecture refers to a modern architectural approach that separates storage from compute, allowing systems to scale more efficiently and flexibly. The idea is inspired by cloud-native database designs where data storage and processing power are decoupled instead of tightly integrated.

In traditional database systems, storage and compute are bundled together. If you need more processing power, you often have to scale the entire system, including storage, even if storage capacity is sufficient. This creates inefficiencies in cost and scalability.

In a NEON-inspired approach, storage and compute operate independently. Storage is centralized and durable, while compute instances can scale up or down based on workload demand. This architecture supports elasticity, better resource utilization, and faster provisioning of environments.

One of the well-known implementations of this concept is Neon, which applies serverless principles to PostgreSQL by separating compute nodes from storage layers. While the term “NEON-inspired” may not refer exclusively to that company, it reflects the broader architectural pattern of disaggregation and cloud-native database design.

Data Virtualization is the virtual consumption layer and connect with anonymous data stores i.e., RDBMS, data appliance, NoSQL, Web Services, SaaS, and enterprise applications etc. As it’s connecting directly with all data stores, the data models of all systems remain intact with no change.

‘It’s considered as one of the solutions for Data Sovereignty problem statement.’

What data stores?

Here data stores are referred as traditional RDBMS or OLTP systems, OLAP systems, Data Warehouse, Data Lake, Delta Lake, Data Vault, Business Vault, Kafka, data appliance, NoSQL, Web Services, SaaS, and enterprise applications etc. Yes, it’s confusing as we all know above data stores serve their purpose for one or the other business requirement then why this another layer on top?

Data Virtualization hosts views connecting directly with all possible data stores. These views can be one-to-one or can be complex SQL written views to pull data from multiple source systems to create a virtualized layer on top of consumer layer to consume.