Knowledge Hub

Data Federation works just like Data Virtualization framework but what is the difference between both.

Relationship between Data Federation and Data Virtualization is just like relationship between Data Mining and Data Science (there is separate topic on Data Mining vs Data Science).

Conceptually, both Data Federation and Data Virtualization works the same way, but Data Federation is restricted to relational database whereas Data Virtualization also includes data appliance, NoSQL, Web Services, SaaS, and enterprise applications etc., as well.

Another major difference i.e., Data Federation doesn’t create a unified data model whereas Data Virtualization does.

Below are a few similar behaviors of both Data Federation and Data Virtualization.

• Single Access Point: Access all data stores from a single environment.
• UAM: Use Access Management at one place.
• Data Quality Management: Data Quality issues can be identified from a single point.
• No Data Movement: No ETL/ ELT is required.
• Ease of use: As users don’t have to write queries in different languages. Users can write in one language, and it will create a wrapper to talk to all the connected systems.
• No Additional Storage: As a virtual copy there is no cost for any separate copy of data, so no additional storage is required.
• Low Risk: As it’s not hosting the data copy as compared to Data Warehouses or Data Lake where data is physically moved, there is no chance of missing any data.
• Better for Analytics: As all the source data models are connected and data is fetched in real like, it makes source data available in real-time for Data Analysts and Data Scientist
• Performance Impact: Some impact on source systems as some queries will run at source.
• Functional Knowledge: Seek dedicated involvement by Data Stewards and Functional consultants.
• No Historical Data: As there is no copy of data hosted so history data is also not maintained. These are best suited along with Data Warehouse in-place.

As organizations matured in their use of Data Warehouses, a new challenge began to surface. Warehouses were excellent for structured, relational data such as sales records, financial transactions, and inventory counts. However, the digital economy was producing entirely new kinds of information, clickstreams, social media posts, sensor readings, videos, documents, and logs. These data types, which fall into the categories of semi-structured and unstructured data, made up the vast majority of enterprise information, yet traditional warehouses were not designed to ingest, store, or process them at scale.

The solution that emerged was the Data Lake. A Data Lake is a centralized repository that stores structured, semi-structured, and unstructured data in its raw form. Unlike a Data Warehouse, which requires data to be modeled and cleansed before it is loaded, a Data Lake follows a schema-on-read philosophy. Data is ingested quickly, without requiring upfront modeling, and structure is applied later when the data is analyzed. This makes the Data Lake highly flexible for exploratory analysis, advanced analytics, and machine learning.

Over the past decade, the terms Big Data and Data Lake have often been used interchangeably, leading to significant confusion in both technical and business conversations. While they are closely related, they are not the same. Understanding the distinction helps organizations make better architectural decisions and avoid unrealistic expectations when planning data initiatives.

Big Data: The Broader Ecosystem

Big Data is not a single technology or storage system. It is the movement and ecosystem that arose to address the challenges posed by the three Vs, volume, velocity, and variety, of modern data. Big Data includes the distributed storage and processing technologies (HDFS, MapReduce, Spark, Kafka), the new types of databases (NoSQL, graph, document, columnar), and the analytical methods that enable organizations to handle massive, fast-moving, and diverse datasets.

In short, Big Data represents the toolbox and paradigm that allowed enterprises to move beyond the limitations of traditional RDBMS and warehouses.

The concept of the Data Lakehouse emerged around 2020 as an architectural response to the limitations of both Data Warehouses and Data Lakes. In simple terms:

Data Warehouse + Data Lake = Data Lakehouse.

Rather than being an entirely new invention, the Lakehouse combines the strengths of existing paradigms, the structured, governed nature of the Data Warehouse and the flexible, large-scale storage of the Data Lake, enhanced by modern innovations in table formats and metadata management.

At its core, the Data Lakehouse addresses two long-standing challenges. Traditional Data Lakes allowed organizations to store vast amounts of raw data but lacked strong transactional guarantees, often leading to inconsistent or unreliable datasets. Data Warehouses, by contrast, provided consistency and governance but were limited in their ability to handle semi-structured and unstructured data at scale. The Lakehouse resolves this by offering:

The Data Hub is a relatively new term in the evolution of enterprise data architecture, appearing after the rise of the Data Lake. It is important to clarify from the outset that a Data Hub is not another type of storage system, nor is it a new analytical framework. Instead, it plays a distinct role as the central integration and distribution point that connects analytical platforms with operational systems.

To understand why Data Hubs matter, it helps to recall the current division of enterprise systems:

• Operational systems (OLTP): These are core enterprise applications such as ERP, CRM, and core banking systems.
• Data Warehouses (DWH): These serve business intelligence needs such as dashboards, reports, and data mining.
• Data Lakes: These enable advanced analytics such as data science, machine learning, and deep learning.

This division creates a practical problem: how can the results of business intelligence and analytics, KPIs, predictions, classifications, and insights, be shared back with the very applications that drive day-to-day business?

The concept of the Data Fabric is often confused with the Data Hub, but while the two share the goal of breaking down silos, they operate at different levels of abstraction. The Data Hub serves as the distribution and operationalization layer, enabling insights from analytical platforms to flow back into enterprise applications. The Data Fabric, by contrast, is a broader architectural paradigm that unifies data governance, integration, and orchestration across the entire enterprise ecosystem.

In the earlier discussion of the Data Hub, we established that enterprise systems are typically divided into three categories:

• Operational systems (OLTP): core business applications such as ERP, CRM, and core banking.
• Data Warehouses (DWH): supporting business intelligence through dashboards, reports, and data mining.
• Data Lakes: enabling advanced analytics, machine learning, and data science.

The Hub emerged to connect these silos and to push the results of BI and analytics back into operational applications.

One of the biggest criticisms of early Data Lakes was their lack of reliability. While they excelled at storing vast volumes of structured, semi-structured, and unstructured data, they struggled to provide the transactional integrity and data consistency that relational databases and warehouses delivered as a matter of course. Operations such as inserts, updates, and deletes, known collectively as DML (Data Manipulation Language), were difficult to perform reliably on file-based storage systems like Hadoop Distributed File System (HDFS) or cloud object stores such as Amazon S3.

To achieve performance and flexibility, many organizations turned to Apache Spark, an in-memory processing engine capable of handling both batch and real-time streaming workloads. However, Spark introduced a different problem: while it enabled fast queries and analytics, it was not ACID-compliant. Without atomicity, consistency, isolation, and durability, organizations faced the risk of incomplete writes, corrupted data, or inconsistencies when jobs failed midway. Workarounds existed, but they were complex and fragile.

Hadoop and its relationship with Big Data. The Big Data ecosystem has many components like NoSQL databases, In-Memory process, MPP, 3 Vs etc. but Hadoop is the backbone, and it was the main reason for the popularity of Big Data. In other words, Hadoop was the Selling Pitch for Salespeople.

'Hadoop is a Java-based framework.' Currently, its latest version is Apache Hadoop 3.3.4.’

Please note, Hadoop is not a solution, it's again an EMPTY storage. One must load data first just like we do in traditional databases before any analytics can be generated. For the first 5-7 years, Sales keep selling Big Data in such a way that you buy Big Data, and it will do magic for you, where all kinds of Analytics will be readily available but no it didn't happen, and you already bought it lolz. Learning: For every new technology, first hire a specialist internally and then invest 😊.

Let's go a little deeper.

Obvious question, difference between Hadoop, HDFS and Hive?

Hadoop is an Open-Source Framework to store structured, semi/unstructured data. Default block size of 128MB (changeable).

HDFS is a Hadoop Files System just like FAT, NTFS, EXT etc.

Hive is only a wrapper on top of files stored in HDFS. Hive behaves just like tables in traditional databases, but it doesn't store any data, it just manages table structures on top of HDFS files. There are 2 ways Hive works, either store files in HDFS directly or create tables which will create HDFS files at the back end. The difference in these approaches is 1) each file stored in HDFS creates its own block so if there are 2 small files e.g., 50 MB each, then there will be 2 blocks created. 2) on the other hand, Hive manages its own blocks so if there are 2 small files of e.g., 50 MB each, then Hive will store both in a single block.

Consider 1000s of 1MB files float in from social media means 1000s of 1MB small blocks created impacting MPP performance. The best practice is to have blocks of 128MB.

Data Serving is also known as the Serving Layer and it has been there since its inception but as technologies have matured, many new channels have been introduced to support this layer.

‘Serving Layer is last component of the Lambda Architecture.’

Lambda Architecture has been explained in separate topic, so we won’t go into details of it but referring to it, as mentioned above, it is the last layer which is exposed to End Users for reporting purposes. This Layer is, if not all the time but majorly designed based on business requirements.

We all know OLTP systems are not meant to be exposed for reporting purposes. In the current era, for reporting, there are many data stores available i.e., Data Warehouse, OLAP, Data Lake, NoSQL, Semantic layers but ideally Semantic Layer should be treated as Serving Layer.

To be more specific

• If Serving Layer is designed for specific business requirement then Semantic Layer is last layer.
• If the Serving Layer is designed for generic use then it can be Data Warehouse, OLAP, Data Lake, NoSQL etc.