Knowledge Hub

Enterprise data architecture is failing. Not because organizations lack investment, or because the problem is inherently too complex, but because the structure required to hold these ecosystems together is either weak or absent. As data landscapes stretch across domains, platforms, and workloads, coherence simply does not sustain itself. Below diagram show the potentiate challenges faced by different persona(s) in an organization.

Over the past decade, there has been no shortage of progress. Cloud-native platforms, lakehouse patterns, distributed processing, governance tooling - none of this is immature anymore. And yet the outcomes tell a different story. Environments expand, but alignment erodes. Governance becomes more visible, more documented, more discussed - but less effective in practice. Analytical capability grows, but trust lags behind, sometimes significantly. These are not isolated execution failures. They point to something more fundamental.

Flow of all Data Modelling Concepts & Techniques

Data Modelling is a foundational concept in the field of data management and analytics. It involves creating different representations of data structures to support various business and technical requirements.

As mentioned, there are around 37 data modelling technique which can be categorized in 5 layers, mentioned below.

1. Business & Enterprise Modelling
2. Semantic & Information Modelling
3. Core Data Modelling
4. Specialized Techniques
5. Governance & AI Extensions

Let’s see how all data modelling techniques get arranged in these layers.

Domain-Driven Design, commonly known as DDD, is an approach to modelling and system design that starts by deeply understanding the business domain. The term domain refers to the problem area the business is working in, such as banking, healthcare, education, or retail. DDD Modelling focuses on capturing the key business concepts, rules, and meanings, and making sure both business people and technical teams talk about them in the same way.

The main goal of DDD Modelling is to ensure that system design closely reflects real business needs. Instead of designing systems based only on technical convenience, DDD encourages teams to model software around how the business actually operates. When done correctly, the system grows naturally alongside the business, rather than becoming disconnected from it.

A central idea in DDD is the creation of a shared or ubiquitous language. This means that the same terms are used consistently by business experts, analysts, and developers. When a business person says “approved loan,” the developer and the analyst understand exactly what that term means, without needing further clarification. This shared language reduces misunderstandings and makes collaboration much easier.

Enterprise Data Modelling is the practice of creating a single, high-level view of data for the entire organization. Instead of looking at data system by system or project by project, it looks across all business areas and defines how key data concepts are understood at an enterprise level. The purpose is to make sure the organization has a shared understanding of its most important data.

The main goal of Enterprise Data Modelling is to provide a consistent framework for managing data across the whole enterprise. When this framework exists, different departments and systems can work together more easily. Data becomes easier to integrate, compare, and trust, because everyone is working from the same definitions.

This modelling approach brings together data from different subject areas such as customers, products, finance, and operations into one unified model. Rather than allowing each system to define data in its own way, Enterprise Data Modelling defines common entities and relationships that apply everywhere. Because of this, it acts as a strategic blueprint rather than a detailed design for a single application.

Conceptual and Logical Architecture Modelling describes the overall structure of an enterprise’s systems and how they interact, without focusing on specific technologies or physical implementations. It provides a blueprint that explains which systems exist, what roles they play, and how data and processes flow between them. At this stage, the goal is understanding and alignment, not technical configuration.

The primary goal of this modelling approach is to ensure that systems, data, and processes are well aligned with business needs. By clearly defining how different parts of the architecture fit together, organizations can design solutions that are scalable, well-integrated, and easier to govern over time. This helps prevent fragmented systems and poorly connected solutions that grow in isolation.

Conceptual and Logical Architecture Modelling focuses on components and interactions, rather than code, databases, or vendors. At the conceptual level, it presents a high-level view of the architecture, typically showing major domains such as applications, data, and infrastructure. This view answers questions like which systems exist and what their responsibilities are. At the logical level, the model goes a step further by showing how data flows between systems, how integrations work, and how systems are logically grouped, still without committing to specific technologies.

In today’s hyper-competitive digital landscape, the imperative for a robust Enterprise Architecture (EA) framework cannot be overstated. For Top Management, synchronizing technological initiatives with overarching business imperatives is essential to optimizing value creation and operational efficacy.

A holistic EA framework initiates with the essential inquiry of "WHY?", the foundational rationale underpinning each initiative. By accurately revealing the purpose and securing explicit management endorsement, each project is thoroughly aligned with strategic imperatives. Comprehensive guidance is then provided on establishing precise Go-Live timelines, delineating resource requisites, and selecting optimal deployment paradigms, whether on-premises or within a Cloud ecosystem.

A robust EA framework encapsulates four pivotal architectural dimensions. Business Architecture is devoted to harmonizing business exigencies with strategic aspirations, facilitating the development of cogent business strategies, models, and processes that engender substantial value for stakeholders. Data Architecture is predicated on the meticulous organization and management of data, fostering the creation of sophisticated data models, definitions, and flows that yield actionable insights and inform strategic decision-making. Application Architecture addresses application requisites, ensuring that enterprise systems and databases are finely tuned for peak performance and functional integrity, thereby augmenting value for end-users. Lastly, Technical Architecture concentrates on the deployment of requisite physical infrastructure, fortified security protocols, and advanced network solutions to support the technical landscape and guarantee the seamless execution of strategic initiatives.

One must first understand the existing Enterprise Data Architecture Landscape as every new project’s data architecture must follow existing Landscape rather introducing new components with every new project. I have personally seen organizations creating Data Components/ Tools Swamp, as in current era of Big Data there are hundreds of tools offering the same functionalities so one must know whether a tool is already there in existing Landscape or not for a particular requirement. I would consider this as, one the biggest Risk in current era.

Data Architecture is the most important aspect of any data related solution or project. As mentioned in the previous chapter, there are four types of architecture i.e., 1) Business Architecture 2) Data Architecture 3) Application Architecture and 4) Technical Architecture. Most of the organizations confuse or mix-up the last three, which is again a recipe for disaster. Data Architecture should remain standalone from Application or Technology Architectures.

Medallion Architecture is a new term but again just like many other jargons used by Salespeople, this one is no more than a Buzz word. In other words, we already know this, we are already use it if we have a lakehouse implemented.

People who are working with Databricks and have implemented their Lakehouse probably know what this Medallion Architecture is. Let me try to explain in a few lines for the audience reading this book.

Medallion Architecture is sitting in Delta Lake tables, and it is about processing data in multiple layers i.e., Bronze ⇒ Silver ⇒ Gold layer tables.

• Raw Layer: This layer is normally not under Medallion Architecture as it totally Raw whereas Bronze, Silver and Gold all are in Delta Lake tables. So Raw Layer hosts source data which can’t be pushed to Bronze Layer directly. Most of the organizations prefer to land all source systems data in Raw layer (I also prefer this).
• Bronze: Those source systems which have connectors with Databricks/ Delta Lake can land data in Bronze else Bronze source is most Raw layer. Some data standardization transformation happens here like.

Framework for Big Data Processing

Lambda Architecture is an essential concept in the world of Big Data, gaining attention as technology for handling massive data volumes has advanced. It forms the backbone of many decision-support systems by providing a structured approach to data processing, integrating both batch and real-time data workflows. The Lambda Architecture model offers a comprehensive system for handling data from generation to end-user access, balancing speed and accuracy through batch and real-time layers.

The Foundations of Lambda Architecture

Lambda Architecture is grounded in lambda calculus, a formal system in mathematical logic that supports computation using function abstraction, application, and variable substitution. Lambda calculus is a universal model that can theoretically simulate any Turing machine, making it foundational to computer science and software engineering. In data processing, Lambda Architecture uses this mathematical basis to create a dual-layered system that can handle data asynchronously and in real time.

Kappa Architecture is mostly confused with Lambda Architecture which is fair as both processes Streaming as well as Batch data. Then how both are different?

Due to high demand of Real-Time data, Kappa Architecture has emerged but not for all kinds of data sizes just like Lambda Architecture is.

We have gone through in detail (in separate topic) like what is Lambda Architecture so won’t go in details like how Streaming and Batch works. Let’s understand how Kappa is different than Lambda Architecture.

Kappa Architecture offers ONLY stream processing using Queuing tools like KAFKA for both Streaming and Batch datasets. It doesn’t offer separate tools or technologies to process batches. It says, process both Real-Time Streaming and Batch data using same tool as KAFKA but the biggest differentiator here is, it stores data in a separate data store which might not be suitable to store TBs or PBs of dataset. It offers batch related downstream applications to connect directly to its storage.