Knowledge Hub

The concept of the Data Lakehouse emerged around 2020 as an architectural response to the limitations of both Data Warehouses and Data Lakes. In simple terms:

Data Warehouse + Data Lake = Data Lakehouse.

Rather than being an entirely new invention, the Lakehouse combines the strengths of existing paradigms, the structured, governed nature of the Data Warehouse and the flexible, large-scale storage of the Data Lake, enhanced by modern innovations in table formats and metadata management.

At its core, the Data Lakehouse addresses two long-standing challenges. Traditional Data Lakes allowed organizations to store vast amounts of raw data but lacked strong transactional guarantees, often leading to inconsistent or unreliable datasets. Data Warehouses, by contrast, provided consistency and governance but were limited in their ability to handle semi-structured and unstructured data at scale. The Lakehouse resolves this by offering:

The Data Hub is a relatively new term in the evolution of enterprise data architecture, appearing after the rise of the Data Lake. It is important to clarify from the outset that a Data Hub is not another type of storage system, nor is it a new analytical framework. Instead, it plays a distinct role as the central integration and distribution point that connects analytical platforms with operational systems.

To understand why Data Hubs matter, it helps to recall the current division of enterprise systems:

• Operational systems (OLTP): These are core enterprise applications such as ERP, CRM, and core banking systems.
• Data Warehouses (DWH): These serve business intelligence needs such as dashboards, reports, and data mining.
• Data Lakes: These enable advanced analytics such as data science, machine learning, and deep learning.

This division creates a practical problem: how can the results of business intelligence and analytics, KPIs, predictions, classifications, and insights, be shared back with the very applications that drive day-to-day business?

The concept of the Data Fabric is often confused with the Data Hub, but while the two share the goal of breaking down silos, they operate at different levels of abstraction. The Data Hub serves as the distribution and operationalization layer, enabling insights from analytical platforms to flow back into enterprise applications. The Data Fabric, by contrast, is a broader architectural paradigm that unifies data governance, integration, and orchestration across the entire enterprise ecosystem.

In the earlier discussion of the Data Hub, we established that enterprise systems are typically divided into three categories:

• Operational systems (OLTP): core business applications such as ERP, CRM, and core banking.
• Data Warehouses (DWH): supporting business intelligence through dashboards, reports, and data mining.
• Data Lakes: enabling advanced analytics, machine learning, and data science.

The Hub emerged to connect these silos and to push the results of BI and analytics back into operational applications.

One of the biggest criticisms of early Data Lakes was their lack of reliability. While they excelled at storing vast volumes of structured, semi-structured, and unstructured data, they struggled to provide the transactional integrity and data consistency that relational databases and warehouses delivered as a matter of course. Operations such as inserts, updates, and deletes, known collectively as DML (Data Manipulation Language), were difficult to perform reliably on file-based storage systems like Hadoop Distributed File System (HDFS) or cloud object stores such as Amazon S3.

To achieve performance and flexibility, many organizations turned to Apache Spark, an in-memory processing engine capable of handling both batch and real-time streaming workloads. However, Spark introduced a different problem: while it enabled fast queries and analytics, it was not ACID-compliant. Without atomicity, consistency, isolation, and durability, organizations faced the risk of incomplete writes, corrupted data, or inconsistencies when jobs failed midway. Workarounds existed, but they were complex and fragile.

As discussed in the topic i.e., Data Warehouse or Data Mart, Star schema and Snowflake Schema both refer to Ralph Kimball Data Warehousing approach.

As per Ralph Kimball, we MUST use Dimensional Modelling for Data Warehousing and Data Marts consisting of Star and Snowflake Schemas.

In Star Schema, the model has a Fact table in the middle and all the Dimensions are joined with the Fact. For example, there is a Fact table called Sales and there can be many dimensions like Time, Location, Product, Categories etc. The Primary Key of the Fact table consists of all the foreign key columns coming from Dimensions. This way when a report or a query is created, there is always one direct join with any Dimension.

As discussed in topic i.e., Data Warehouse or Data Mart that Star schema and Snowflake Schema both refer to Ralph Kimball Data Warehousing approach.

As per Ralph Kimball, we MUST use Dimensional Modelling for Data Warehousing and Data Marts consisting of Star and Snowflake Schemas.

In Snowflake Schema, the model also has a Fact table in the middle and Dimensions around it but here Dimensions can be normalized further in multiple Sub-dimensions which changes the query to create a report where there will be multiple joins to reach a conclusive result. For example, there is a Fact table called Sales and a Dimension i.e., Location, this location Dimension can be normalized further in State, City, and Postal Code etc.

As discussed in topic i.e., Data Warehouse or Data Mart that Star schema and Snowflake Schema both refer to Ralph Kimball Data Warehousing approach. But what about Galaxy Schema?

You must be wondering that we have been hearing about Facts and Dimensions in both Star and Snowflake schema(s) but where are Conformed Dimensions, so here it is.

Galaxy Schema is also known as Fact Constellation Schema and is more complex than Star and Snowflake schema. It’s further divide Star Schema into small Star Schema(s) where there are more than one Fact Table and have Reusable Dimension to connect multiple Fact Tables.

These dimensions which can connect with multiple Fact tables across multiple Star Schema(s) are known as Conformed Dimensions.

Cubes are a very controversial element of OLAP world where most of the folks think, it not worth having those.

Cubes host data from facts and dimensions in one table so query can be super-fast, in Cubes multi-dimensional data is pre-loaded.

Disadvantages

• These are huge in size.
• Very difficult to refresh in real time.
• Housekeeping is a challenge.

Below is Star or Snowflake schema representation. When data is joined from Fact and Dimension and loaded on one table it is called an OLAP Cube.

There are multiple types of OLAP i.e.  ROLAP, MOLAP, HOLAP, HTAP, WOLAP, DOLAP and SOLAP.

• ROLAP [Relational On-Line Analytical Processing]: It sits directly on top of RDBMS which can be Star/ Snowflake Schema(s) and can be built using standard SQL. Another benefit, ROLAP doesn’t need additional storage as it runs on top of RDBMS.
• MOLAP [Multi-dimensional OLAP]: Cubes are the best example of MOLAP where multi-dimensional data can be loaded in a separate storage but in an extremely fast manner. Using Cubes, users can slice, dice, and drill up/ down, roll up and pivot the data within the same one cube eliminating the need to join with other tables. If a Cube has more than 3 dimensions of data in it, it’s called Hypercube.
• HOLAP [Hybrid]: This is a mixture of ROLAP or MOLAP.
• HTAP [Hybrid Transaction / Analytical Processing]: This is for in-memory data systems that do both online transaction processing (OLTP) and online analytical processing (OLAP).
• WOLAP [Web-enabled]: This refers to the only Web-Based OLAP with no connectivity with on-premises and is utilized through the internet browser.
• DOLAP [Desktop On-Line Analytical Processing]: In this OLAP, users can download a portion of OLAP on their machine for offline working.
• SOLAP [Spatial]: This is to tackle issues in the field of Geo-Business Intelligence.

Before I came into the Big Data domain, in early 2010, my assumption was, standard SQL will not work for NoSQL databases, but I was 180 degrees wrong. NoSQL means Not-Only-SQL. Yeah, I felt dumb after knowing this lolz 😊.

In the past, we had only one Database Type i.e., RDBMS which here is referred to as SQL but with the introduction of Big Data, 4 more types of NoSQL databases came onto the surface i.e., Key-Value, Columnar, Document and Graph DB.

Examples of SQL database types are Oracle, DB2, Teradata, SQL Server, MySQL etc. which we all have been using for decades.

Hope we are on the same page that all SQL/RDBMS databases are just empty storage. The same is true with all these new NoSQL databases; these are again empty storages. In contrast with SQL/RDBMS databases which use server OS file systems like EXT, FAT21 etc. the new NoSQL database can additionally use HDFS.

Now, there is big confusion for storage selection in Hadoop as for a normal person, there are 5 new data stores in Hadoop i.e., 1) Hive 2) Key-Value DB (NoSQL) 3) Columnar DB (NoSQL) 4) Document DB (NoSQL) and 5) Graph DB (NoSQL). Yes, there are 5 data stores but there is 1 storage that is HDFS Hadoop Files Storage.

Data Mesh is normally confused with Data Mashup (explained separately), but both are totally different.

Data Mesh is a modern approach to data architecture and governance that emphasizes decentralization and domain-driven ownership of data products. Unlike traditional centralized data systems, which rely on a monolithic team to manage and deliver data assets across an organization, Data Mesh shifts the focus towards empowering individual domains to become autonomous in managing their data products. This paradigm fosters scalability, improves data usability, and ensures alignment with specific business needs.

Understanding the Foundations of Data Mesh

At its core, Data Mesh revolves around four key principles:

1. Domain-Oriented Decentralization: The fundamental premise of Data Mesh is the distribution of data ownership to individual domains. A domain, in this context, refers to a specific business area or operational unit, such as marketing, sales, or customer service. Each domain is responsible for defining, managing, and maintaining its own data products. By doing so, Data Mesh reduces the bottlenecks and inefficiencies associated with centralized data teams.
2. Data as a Product (DAaP): Data Mesh introduces the concept of treating data as a product. This means that data products must be designed with the same rigor and customer-centricity as any software product. Each data product must: