Knowledge Hub

Please note, AAaaS has nothing to do with Cloud, it can be on On-Premises or On-Cloud. AAaaS is an environment setup for Data Scientists to play around with Machine Learning and Deep Learning Modelling.

Advance Analytics as a Service (AAaaS) is a new concept and still maturing, due to which it is still very confusing. Different companies are offering different approaches, so there is no one best approach.

Two approaches been followed.

1. AAaaS as PaaS: Modelling tools like Databricks, Azure AML etc., are available as pre-installed services (refer to topic: PaaS) where customers can provision these tools in few clicks and simply start developing their Machine Learning or Deep Learning Models to build their own solutions. Here, customers must have their own MLOps along with CI/CD.
2. AAaaS as SaaS: In this approach, Solution Providers hosts pre-built or OOTB Machine Learning and Deep Learning Models. These are offered to customers to consume only. Customers inject their own dataset for relevant data analysis and analytics. Here, customers must follow Solution Providers MLOps along with CI/CD.

Data Virtualization is the virtual consumption layer and connect with anonymous data stores i.e., RDBMS, data appliance, NoSQL, Web Services, SaaS, and enterprise applications etc. As it’s connecting directly with all data stores, the data models of all systems remain intact with no change.

‘It’s considered as one of the solutions for Data Sovereignty problem statement.’

What data stores?

Here data stores are referred as traditional RDBMS or OLTP systems, OLAP systems, Data Warehouse, Data Lake, Delta Lake, Data Vault, Business Vault, Kafka, data appliance, NoSQL, Web Services, SaaS, and enterprise applications etc. Yes, it’s confusing as we all know above data stores serve their purpose for one or the other business requirement then why this another layer on top?

Data Virtualization hosts views connecting directly with all possible data stores. These views can be one-to-one or can be complex SQL written views to pull data from multiple source systems to create a virtualized layer on top of consumer layer to consume.

Data Federation works just like Data Virtualization framework but what is the difference between both.

Relationship between Data Federation and Data Virtualization is just like relationship between Data Mining and Data Science (there is separate topic on Data Mining vs Data Science).

Conceptually, both Data Federation and Data Virtualization works the same way, but Data Federation is restricted to relational database whereas Data Virtualization also includes data appliance, NoSQL, Web Services, SaaS, and enterprise applications etc., as well.

Another major difference i.e., Data Federation doesn’t create a unified data model whereas Data Virtualization does.

Below are a few similar behaviors of both Data Federation and Data Virtualization.

• Single Access Point: Access all data stores from a single environment.
• UAM: Use Access Management at one place.
• Data Quality Management: Data Quality issues can be identified from a single point.
• No Data Movement: No ETL/ ELT is required.
• Ease of use: As users don’t have to write queries in different languages. Users can write in one language, and it will create a wrapper to talk to all the connected systems.
• No Additional Storage: As a virtual copy there is no cost for any separate copy of data, so no additional storage is required.
• Low Risk: As it’s not hosting the data copy as compared to Data Warehouses or Data Lake where data is physically moved, there is no chance of missing any data.
• Better for Analytics: As all the source data models are connected and data is fetched in real like, it makes source data available in real-time for Data Analysts and Data Scientist
• Performance Impact: Some impact on source systems as some queries will run at source.
• Functional Knowledge: Seek dedicated involvement by Data Stewards and Functional consultants.
• No Historical Data: As there is no copy of data hosted so history data is also not maintained. These are best suited along with Data Warehouse in-place.

The term Big Data emerged in the early 2000s as enterprises began facing unprecedented growth in the amount, speed, and diversity of information being generated. Traditional databases and warehouses, while reliable for structured data, were not designed to handle this new reality. Big Data describes not only the massive scale of data itself but also the ecosystem of technologies, methods, and practices developed to capture, store, and analyze it.

At the heart of Big Data is the concept of the “three Vs”, later extended to four or more:

• Volume: the enormous scale of data generated from transactions, machines, sensors, mobile devices, and social platforms.
• Velocity: the speed at which data arrives, often requiring real-time or near real-time processing instead of traditional batch updates.
• Variety: the diversity of data formats, structured (tables), semi-structured (JSON, XML), and unstructured (text, images, videos, audio, logs).
• Veracity (sometimes added): the need to ensure trustworthiness and quality despite scale and diversity.

As organizations matured in their use of Data Warehouses, a new challenge began to surface. Warehouses were excellent for structured, relational data such as sales records, financial transactions, and inventory counts. However, the digital economy was producing entirely new kinds of information, clickstreams, social media posts, sensor readings, videos, documents, and logs. These data types, which fall into the categories of semi-structured and unstructured data, made up the vast majority of enterprise information, yet traditional warehouses were not designed to ingest, store, or process them at scale.

The solution that emerged was the Data Lake. A Data Lake is a centralized repository that stores structured, semi-structured, and unstructured data in its raw form. Unlike a Data Warehouse, which requires data to be modeled and cleansed before it is loaded, a Data Lake follows a schema-on-read philosophy. Data is ingested quickly, without requiring upfront modeling, and structure is applied later when the data is analyzed. This makes the Data Lake highly flexible for exploratory analysis, advanced analytics, and machine learning.

Over the past decade, the terms Big Data and Data Lake have often been used interchangeably, leading to significant confusion in both technical and business conversations. While they are closely related, they are not the same. Understanding the distinction helps organizations make better architectural decisions and avoid unrealistic expectations when planning data initiatives.

Big Data: The Broader Ecosystem

Big Data is not a single technology or storage system. It is the movement and ecosystem that arose to address the challenges posed by the three Vs, volume, velocity, and variety, of modern data. Big Data includes the distributed storage and processing technologies (HDFS, MapReduce, Spark, Kafka), the new types of databases (NoSQL, graph, document, columnar), and the analytical methods that enable organizations to handle massive, fast-moving, and diverse datasets.

In short, Big Data represents the toolbox and paradigm that allowed enterprises to move beyond the limitations of traditional RDBMS and warehouses.

The concept of the Data Lakehouse emerged around 2020 as an architectural response to the limitations of both Data Warehouses and Data Lakes. In simple terms:

Data Warehouse + Data Lake = Data Lakehouse.

Rather than being an entirely new invention, the Lakehouse combines the strengths of existing paradigms, the structured, governed nature of the Data Warehouse and the flexible, large-scale storage of the Data Lake, enhanced by modern innovations in table formats and metadata management.

At its core, the Data Lakehouse addresses two long-standing challenges. Traditional Data Lakes allowed organizations to store vast amounts of raw data but lacked strong transactional guarantees, often leading to inconsistent or unreliable datasets. Data Warehouses, by contrast, provided consistency and governance but were limited in their ability to handle semi-structured and unstructured data at scale. The Lakehouse resolves this by offering:

The Data Hub is a relatively new term in the evolution of enterprise data architecture, appearing after the rise of the Data Lake. It is important to clarify from the outset that a Data Hub is not another type of storage system, nor is it a new analytical framework. Instead, it plays a distinct role as the central integration and distribution point that connects analytical platforms with operational systems.

To understand why Data Hubs matter, it helps to recall the current division of enterprise systems:

• Operational systems (OLTP): These are core enterprise applications such as ERP, CRM, and core banking systems.
• Data Warehouses (DWH): These serve business intelligence needs such as dashboards, reports, and data mining.
• Data Lakes: These enable advanced analytics such as data science, machine learning, and deep learning.

This division creates a practical problem: how can the results of business intelligence and analytics, KPIs, predictions, classifications, and insights, be shared back with the very applications that drive day-to-day business?

The concept of the Data Fabric is often confused with the Data Hub, but while the two share the goal of breaking down silos, they operate at different levels of abstraction. The Data Hub serves as the distribution and operationalization layer, enabling insights from analytical platforms to flow back into enterprise applications. The Data Fabric, by contrast, is a broader architectural paradigm that unifies data governance, integration, and orchestration across the entire enterprise ecosystem.

In the earlier discussion of the Data Hub, we established that enterprise systems are typically divided into three categories:

• Operational systems (OLTP): core business applications such as ERP, CRM, and core banking.
• Data Warehouses (DWH): supporting business intelligence through dashboards, reports, and data mining.
• Data Lakes: enabling advanced analytics, machine learning, and data science.

The Hub emerged to connect these silos and to push the results of BI and analytics back into operational applications.

One of the biggest criticisms of early Data Lakes was their lack of reliability. While they excelled at storing vast volumes of structured, semi-structured, and unstructured data, they struggled to provide the transactional integrity and data consistency that relational databases and warehouses delivered as a matter of course. Operations such as inserts, updates, and deletes, known collectively as DML (Data Manipulation Language), were difficult to perform reliably on file-based storage systems like Hadoop Distributed File System (HDFS) or cloud object stores such as Amazon S3.

To achieve performance and flexibility, many organizations turned to Apache Spark, an in-memory processing engine capable of handling both batch and real-time streaming workloads. However, Spark introduced a different problem: while it enabled fast queries and analytics, it was not ACID-compliant. Without atomicity, consistency, isolation, and durability, organizations faced the risk of incomplete writes, corrupted data, or inconsistencies when jobs failed midway. Workarounds existed, but they were complex and fragile.