Knowledge Hub

Domain-Driven Design, commonly known as DDD, is an approach to modelling and system design that starts by deeply understanding the business domain. The term domain refers to the problem area the business is working in, such as banking, healthcare, education, or retail. DDD Modelling focuses on capturing the key business concepts, rules, and meanings, and making sure both business people and technical teams talk about them in the same way.

The main goal of DDD Modelling is to ensure that system design closely reflects real business needs. Instead of designing systems based only on technical convenience, DDD encourages teams to model software around how the business actually operates. When done correctly, the system grows naturally alongside the business, rather than becoming disconnected from it.

A central idea in DDD is the creation of a shared or ubiquitous language. This means that the same terms are used consistently by business experts, analysts, and developers. When a business person says “approved loan,” the developer and the analyst understand exactly what that term means, without needing further clarification. This shared language reduces misunderstandings and makes collaboration much easier.

Data Mesh is normally confused with Data Mashup (explained separately), but both are totally different.

Data Mesh is a modern approach to data architecture and governance that emphasizes decentralization and domain-driven ownership of data products. Unlike traditional centralized data systems, which rely on a monolithic team to manage and deliver data assets across an organization, Data Mesh shifts the focus towards empowering individual domains to become autonomous in managing their data products. This paradigm fosters scalability, improves data usability, and ensures alignment with specific business needs.

Understanding the Foundations of Data Mesh

At its core, Data Mesh revolves around four key principles:

1. Domain-Oriented Decentralization: The fundamental premise of Data Mesh is the distribution of data ownership to individual domains. A domain, in this context, refers to a specific business area or operational unit, such as marketing, sales, or customer service. Each domain is responsible for defining, managing, and maintaining its own data products. By doing so, Data Mesh reduces the bottlenecks and inefficiencies associated with centralized data teams.
2. Data as a Product (DAaP): Data Mesh introduces the concept of treating data as a product. This means that data products must be designed with the same rigor and customer-centricity as any software product. Each data product must:

HTAP stands for Hybrid Transactional/ Analytical Processing. It is a new architecture proposed by Garner in 2014.

In previous topics, we have established that there is an OLTP database for operational tasks and then Data Warehouse, Data Lake, Delta Lake, Data Lakehouse, Data/ Business Vault etc. for analysis and analysis. All keep data separately in their own storage for management decisions.

HTAP is proposing to use one system for both transaction and analytical workload. It is offering an in-memory plus columnar approach for analytical processes.

With the advancements in hardware, in-memory and Cloud, Gartner thinks it is very much possible to have both Transactional and Analytical processing on the same database.

Advantages

• No duplication of data as data will not move from the same one operational database.
• Business or Management can make decisions on actual data in real-time, in contrast if data must move from OLTP to OLAP systems.

As per the name depiction, Data Locality looks like knowing the location of the data; it is partially relevant.

Traditionally, to use data for analysis, at the backend the data is fetched from storage/ data nodes into master nodes, for computation. There is a complete history of how initially whole data was pulled then blocks were pulled then the columnar pull concept came-in, etc. so we won't go into detail about how technology has evolved to process data.

Since semi and unstructured data has become a necessary part of the Data World, it comes with huge storage requirements i.e., in TB and even in PB. To host this humongous data, clusters of 100s of storage/ data nodes are set up, so it is not possible to pull this size of data for computation in master nodes which are normally few. It was a critical requirement to process queries where data is residing.

With the introduction of Big Data/ Hadoop/ HDFS where data is stored in the shape of blocks, the concept surfaced i.e., instead of fetching data for computation, let's take the query to the storage/ data nodes, process it there and fetch only processed data. This process of moving the computation close to where the actual data resides on the storage/ data node, instead of moving large data to computation, is called Data Locality.

There are four types of instructions to pull data i.e., SISD, SIMD, MISD and MIMD.

• SISD: Single Instruction Stream and Single Data Stream
• SIMD: Single Instruction Stream and Multiple Data Stream
• MISD: Multiple Instruction Stream and Single Data Stream
• MIMD: Multiple Instruction Stream and Multiple Data Stream

Before we jump into understanding all above mentioned techniques, let’s first understand what is a Core in a machine or in a computer or in a server etc.?

A CPU Core is built of billions of microscopic transistors to process instructions passed to it. One instruction is handled by One core, if someone needs multitasking or multiple instruction in parallel, minimum 2 cores are required.

We know when we insert or load data in a computer or server or in a node, it gets saved in hard disk. We won’t go in to details how it is stored on the hard disk. We can imagine when we write something on paper, consider the same way, data is saved in a hard disk.

To succeed with metadata management, you will need an Operational Reference Model. It is the way to describe the structures and relationships of your business metadata.

The Role and relationship of an Operational Reference Model

Conceptual models, logical models, physical models and canonical models have a specific purpose which is either communication and understanding, or representation of an individual system. Operational Reference Models come into play when organizations require an operational layer which bridges multiple systems and functions and provides a common point of reference or shared language for the enterprise. The Operational Reference Model is similar. to an anti-corruption layer and a semantic layer in its purpose but with an important difference, it's owned and governed by your subject matter experts.

Operational reference model governance

An operational reference model is at its core a set of metadata structured with the help of taxonomies and ontologies. Each part, each domain, is governed by the subject matter experts in the organization in accordance with a federated governance approach.

A Unified Approach to Data Ecosystem Management

The concept of Data Platform Gravity revolves around a holistic framework where every aspect of data handling, from ingestion to transformation, analytics, and deployment, is achieved within a single, unified platform. Inspired by the Data Fabric paradigm, it advocates for seamless operations across the entire data lifecycle, reducing complexities, minimizing data silos, and empowering organizations to derive value efficiently and effectively.

Understanding Data Platform Gravity

In a traditional data management ecosystem, disparate tools and technologies often create bottlenecks. A single project might involve different platforms for:

• Extracting and loading data from sources,
• Transforming and enriching data,
• Performing analytics and data science experiments,
• Visualizing results in dashboards or reports, and
• Sharing data insights with external systems or stakeholders.

In today’s rapidly evolving technological landscape, the biggest challenge organizations face is, data integration and maintainability. With the exponential growth of data and the widespread adoption of AI technologies, businesses are increasingly grappling with the need to manage both effectively. Unfortunately, many traditional architectures treat data and AI as distinct entities, leading to a host of challenges that hinder an organization’s ability to innovate, scale, and adapt.

One of the primary issues with the traditional approach is, the fragmentation of data and AI systems into separate silos. Data is often stored and managed in one architecture, while AI models are built, trained, and deployed in another. This separation creates significant friction points.

To overcome these challenges, organizations must embrace a more integrated approach that treats data and AI as interconnected components of a single architecture.