Knowledge Hub

A Logical Data Warehouse (LDW) is not a new storage technology but an architectural approach to delivering a unified data warehouse view without physically consolidating all data into one centralized repository. Unlike a traditional physical data warehouse, which extracts, transforms, and loads data into a dedicated storage layer, a Logical Data Warehouse integrates data virtually. It connects distributed data sources and presents them as a single, consistent analytical environment.

The concept does not replace traditional modeling approaches. A Logical Data Warehouse can still support third normal form structures aligned with the philosophy of Bill Inmon or dimensional modeling approaches such as star and snowflake schemas aligned with Ralph Kimball. The distinction lies not in the modeling technique but in the physical location of the data. In an LDW, data may remain in source systems, data lakes, physical warehouses, business vault layers, delta-based platforms, streaming systems, or APIs. The integration happens logically rather than physically.

As organizations matured in their use of Data Warehouses, a new challenge began to surface. Warehouses were excellent for structured, relational data such as sales records, financial transactions, and inventory counts. However, the digital economy was producing entirely new kinds of information, clickstreams, social media posts, sensor readings, videos, documents, and logs. These data types, which fall into the categories of semi-structured and unstructured data, made up the vast majority of enterprise information, yet traditional warehouses were not designed to ingest, store, or process them at scale.

The solution that emerged was the Data Lake. A Data Lake is a centralized repository that stores structured, semi-structured, and unstructured data in its raw form. Unlike a Data Warehouse, which requires data to be modeled and cleansed before it is loaded, a Data Lake follows a schema-on-read philosophy. Data is ingested quickly, without requiring upfront modeling, and structure is applied later when the data is analyzed. This makes the Data Lake highly flexible for exploratory analysis, advanced analytics, and machine learning.

Over the past decade, the terms Big Data and Data Lake have often been used interchangeably, leading to significant confusion in both technical and business conversations. While they are closely related, they are not the same. Understanding the distinction helps organizations make better architectural decisions and avoid unrealistic expectations when planning data initiatives.

Big Data: The Broader Ecosystem

Big Data is not a single technology or storage system. It is the movement and ecosystem that arose to address the challenges posed by the three Vs, volume, velocity, and variety, of modern data. Big Data includes the distributed storage and processing technologies (HDFS, MapReduce, Spark, Kafka), the new types of databases (NoSQL, graph, document, columnar), and the analytical methods that enable organizations to handle massive, fast-moving, and diverse datasets.

In short, Big Data represents the toolbox and paradigm that allowed enterprises to move beyond the limitations of traditional RDBMS and warehouses.

The concept of the Data Lakehouse emerged around 2020 as an architectural response to the limitations of both Data Warehouses and Data Lakes. In simple terms:

Data Warehouse + Data Lake = Data Lakehouse.

Rather than being an entirely new invention, the Lakehouse combines the strengths of existing paradigms, the structured, governed nature of the Data Warehouse and the flexible, large-scale storage of the Data Lake, enhanced by modern innovations in table formats and metadata management.

At its core, the Data Lakehouse addresses two long-standing challenges. Traditional Data Lakes allowed organizations to store vast amounts of raw data but lacked strong transactional guarantees, often leading to inconsistent or unreliable datasets. Data Warehouses, by contrast, provided consistency and governance but were limited in their ability to handle semi-structured and unstructured data at scale. The Lakehouse resolves this by offering:

As discussed in the topic i.e., Data Warehouse or Data Mart, Star schema and Snowflake Schema both refer to Ralph Kimball Data Warehousing approach.

As per Ralph Kimball, we MUST use Dimensional Modelling for Data Warehousing and Data Marts consisting of Star and Snowflake Schemas.

In Star Schema, the model has a Fact table in the middle and all the Dimensions are joined with the Fact. For example, there is a Fact table called Sales and there can be many dimensions like Time, Location, Product, Categories etc. The Primary Key of the Fact table consists of all the foreign key columns coming from Dimensions. This way when a report or a query is created, there is always one direct join with any Dimension.

As discussed in topic i.e., Data Warehouse or Data Mart that Star schema and Snowflake Schema both refer to Ralph Kimball Data Warehousing approach.

As per Ralph Kimball, we MUST use Dimensional Modelling for Data Warehousing and Data Marts consisting of Star and Snowflake Schemas.

In Snowflake Schema, the model also has a Fact table in the middle and Dimensions around it but here Dimensions can be normalized further in multiple Sub-dimensions which changes the query to create a report where there will be multiple joins to reach a conclusive result. For example, there is a Fact table called Sales and a Dimension i.e., Location, this location Dimension can be normalized further in State, City, and Postal Code etc.

As discussed in topic i.e., Data Warehouse or Data Mart that Star schema and Snowflake Schema both refer to Ralph Kimball Data Warehousing approach. But what about Galaxy Schema?

You must be wondering that we have been hearing about Facts and Dimensions in both Star and Snowflake schema(s) but where are Conformed Dimensions, so here it is.

Galaxy Schema is also known as Fact Constellation Schema and is more complex than Star and Snowflake schema. It’s further divide Star Schema into small Star Schema(s) where there are more than one Fact Table and have Reusable Dimension to connect multiple Fact Tables.

These dimensions which can connect with multiple Fact tables across multiple Star Schema(s) are known as Conformed Dimensions.

Cubes are a very controversial element of OLAP world where most of the folks think, it not worth having those.

Cubes host data from facts and dimensions in one table so query can be super-fast, in Cubes multi-dimensional data is pre-loaded.

Disadvantages

• These are huge in size.
• Very difficult to refresh in real time.
• Housekeeping is a challenge.

Below is Star or Snowflake schema representation. When data is joined from Fact and Dimension and loaded on one table it is called an OLAP Cube.

There are multiple types of OLAP i.e.  ROLAP, MOLAP, HOLAP, HTAP, WOLAP, DOLAP and SOLAP.

• ROLAP [Relational On-Line Analytical Processing]: It sits directly on top of RDBMS which can be Star/ Snowflake Schema(s) and can be built using standard SQL. Another benefit, ROLAP doesn’t need additional storage as it runs on top of RDBMS.
• MOLAP [Multi-dimensional OLAP]: Cubes are the best example of MOLAP where multi-dimensional data can be loaded in a separate storage but in an extremely fast manner. Using Cubes, users can slice, dice, and drill up/ down, roll up and pivot the data within the same one cube eliminating the need to join with other tables. If a Cube has more than 3 dimensions of data in it, it’s called Hypercube.
• HOLAP [Hybrid]: This is a mixture of ROLAP or MOLAP.
• HTAP [Hybrid Transaction / Analytical Processing]: This is for in-memory data systems that do both online transaction processing (OLTP) and online analytical processing (OLAP).
• WOLAP [Web-enabled]: This refers to the only Web-Based OLAP with no connectivity with on-premises and is utilized through the internet browser.
• DOLAP [Desktop On-Line Analytical Processing]: In this OLAP, users can download a portion of OLAP on their machine for offline working.
• SOLAP [Spatial]: This is to tackle issues in the field of Geo-Business Intelligence.

These are one of the most confusing terms in the data domain. Both are mostly confused with each other.

The major difference between both is.

• Data Mining is done on ONLY structured data.
• Data Science is done on structured, semi and unstructured data.

Data Science is mostly tagged with Big Data and Data Mining is tagged with Data Warehousing. There are many definitions and comparisons for both in the market as both do machine learning, deep learning, data cleansing, statics, and math algorithms etc.

‘Data Science is the Superset of Data Mining.’

Data Mining is also known as KDD (Knowledge Discovery in Data). First it was performed in excel, then with the introduction of DWH, an automated process started discovering historical patterns and other valuable information from large data sets which are structured.

Now with the introduction of Big Data, Data Lake, NoSQL databases, semi and unstructured datasets, the magnitude of the data size has taken a huge step ahead and Data Science has come onto the surface.

We touched base with Data Mining and Data Science, it will be ideal to discuss Data Dredging, Snooping, p-hacking, and Fishing. New terms right 😊?

Data Dredging, Snooping and Fishing all refer to the same behavior of data analysis BUT without proper hypothesis and relationship among datasets.

Data mining finds results based on the correlation of data in large data sets, but Data Dredging, Snooping, p-hacking, and Fishing find results based on chance methodology. In other words, Data Dredging, Snooping, p-hacking, and Fishing share the results which require more investigation. These sometimes bypass Data Mining techniques and come up with immature conclusions. On occasions, it shows more details about something than it contains.

There are two types of Fact Findings in any analysis that ultimately assists in DSS i.e., 1) known facts 2) unknown facts. Data mining techniques extract Known Facts as it’s done only on structured datasets and mostly, we know what's there in structured data like relationships between tables and relationships of data within the same table. But that's Not True, sometimes, even in structured data there are Unknown Facts and that’s where Data Dredging, Snooping and Fishing techniques come into the action.