Knowledge Hub

Data Subsetting is as simple as taking a small piece of data, with referential integrity in place, for some activity. For example, there is a requirement to build a data science model and it requires data for users aged between 25-35 from X Location having certain buying/ spending patterns. For this requirement, a particular dataset will be extracted for the data science development team to build and train their data models.

Don't confuse Data Subsetting with Data Masking. Data Masking is still required on this smaller piece of the dataset from a security point of view. Having said that, conducting a Data Subsetting exercise is not as simple. One first must understand the functional requirements of the use case, then one must also understand what data is there in the database as whole. Wrongs or partial Data Subsetting will end up producing wrong data model results and will damage trust in data by decision-makers.

This term is normally known by the Security department and is tagged with Hackers.

‘Data Scraping is also known as Web Scraping.’

The minor difference between both terms is Data scraping means searching the location of the data and then extracting it. In other words, it does not copy and paste rather extract data in a precise and accurate manner whereas Web scraping involves extracting exactly everything from web pages.

Data Scraping or Web Scraping is the process of importing information from a website into a spreadsheet or local file saved on your computer.

Popular uses of data scraping include:

• Research for web content/business intelligence.
• Pricing for travel booker sites/price comparison sites
• Finding sales leads/conducting market research by crawling public data sources (e.g., Yell and Twitter)
• Sending product data from an e-commerce site to another online vendor (e.g., Google Shopping)

It allows us to take unstructured, scattered data from multiple sources and collect it in one place and make it structured for better use.

In the current era there are many data stores like Database, Data Warehouse, Data Marts, Big Data, Data Lake, Data Vault, Business Vault, SQL, and NoSQL etc. Use of all these terms is different with their own horizon, their own skill sets whereas tools and languages can be the same for all of these.

First one must understand is that Database is a datastore, we can’t link it to any data modelling solution. In other words, OLTP or Data Warehouse or Data Mart or OLAP all use databases as a medium to store their datasets. Additionally, Databases can be SQL or NoSQL.

Now, OLTP is on one side of the line and Data Warehouse/ Data Marts and Big Data/ Data Lake etc. are on the other side. Normally, databases are called OLTP (Online Transactional Processing) whereas Data Warehouse/ Data Marts fall under the term OLAP (Online Analytical Processing).

As far as Big Data/ Data Lake is concerned, Data World has yet to tag these with either OLTP or OLAP but for sure these are not meant for OLTP. At the same time NoSQL is stepping into OLTP zone and is supporting OLTP as well.

The idea of a Data Warehouse emerged when organizations realized that transactional systems (OLTP), while excellent for recording day-to-day operations, were never designed for heavy reporting or analytical workloads. As business leaders demanded consolidated views of performance, finance, sales, and customer behaviour, data modelers introduced the concept of a centralized analytical datastore, the Data Warehouse, to relieve OLTP systems from the burden of reporting and to provide trusted, historical insight.

It is important to clarify a common source of confusion: a Data Warehouse is not the same as Dimensional Modelling. The warehouse refers to the central repository that consolidates data from multiple source systems, while dimensional modelling is only one possible technique for structuring that repository. A Data Warehouse can be designed in different ways, depending on organizational priorities such as performance, scalability, or agility.

This topic is very close to many experienced data modelers because it is surprisingly misunderstood. The confusion usually begins with one assumption: if both systems use Third Normal Form (3NF), then they must be conceptually the same. They are not.

In data architecture, the Data Warehouse concept popularized by Bill Inmon embraces the use of 3NF. Since 3NF is also widely used in OLTP systems, many architects assume the modeling philosophy and operational behavior are identical. The structure may look similar, but the intent, behavior, and lifecycle of data are fundamentally different.

OLTP 3NF: Designed for Transactional Efficiency

In an OLTP (Online Transaction Processing) system, the objective is operational efficiency. The system must support high-volume, real-time transactions such as order placement, payments, inventory updates, and customer modifications.

The primary goals are:

• Fast inserts, updates, and deletes
• Concurrency control
• Storage optimization
• Data consistency for current state

Immutable Data Warehouse has come from immutable data. Data folks are aware of it but don’t use it more often.

‘Immutable means not changeable.’

Now, the question is how it’s relevant to us in Data World. We know when we are inserting new data in a database, at the same time we also have the option to change it by using update or delete commands which means data is mutable.

Mutable feature is a MUST TO HAVE in any database at a technology level, means, any databases or data store MUST allow to update or delete. One should or should not update or delete, can be a business requirement but cannot be technology limitation.

‘By default, Data Warehouse is an Immutable datastore.’

Immutable feature is not only about, not able to update or delete data, it’s about which data should be immutable, and which should be mutable. For this, one must understand the data.

For example, CUSTOMER dataset contains many types of data e.g., name, date of birth, parents, children, address, contact details, jobs, cars etc. In this dataset, you must have judged which information is immutable and which mutable.

A Logical Data Warehouse (LDW) is not a new storage technology but an architectural approach to delivering a unified data warehouse view without physically consolidating all data into one centralized repository. Unlike a traditional physical data warehouse, which extracts, transforms, and loads data into a dedicated storage layer, a Logical Data Warehouse integrates data virtually. It connects distributed data sources and presents them as a single, consistent analytical environment.

The concept does not replace traditional modeling approaches. A Logical Data Warehouse can still support third normal form structures aligned with the philosophy of Bill Inmon or dimensional modeling approaches such as star and snowflake schemas aligned with Ralph Kimball. The distinction lies not in the modeling technique but in the physical location of the data. In an LDW, data may remain in source systems, data lakes, physical warehouses, business vault layers, delta-based platforms, streaming systems, or APIs. The integration happens logically rather than physically.

The term Big Data emerged in the early 2000s as enterprises began facing unprecedented growth in the amount, speed, and diversity of information being generated. Traditional databases and warehouses, while reliable for structured data, were not designed to handle this new reality. Big Data describes not only the massive scale of data itself but also the ecosystem of technologies, methods, and practices developed to capture, store, and analyze it.

At the heart of Big Data is the concept of the “three Vs”, later extended to four or more:

• Volume: the enormous scale of data generated from transactions, machines, sensors, mobile devices, and social platforms.
• Velocity: the speed at which data arrives, often requiring real-time or near real-time processing instead of traditional batch updates.
• Variety: the diversity of data formats, structured (tables), semi-structured (JSON, XML), and unstructured (text, images, videos, audio, logs).
• Veracity (sometimes added): the need to ensure trustworthiness and quality despite scale and diversity.

As organizations matured in their use of Data Warehouses, a new challenge began to surface. Warehouses were excellent for structured, relational data such as sales records, financial transactions, and inventory counts. However, the digital economy was producing entirely new kinds of information, clickstreams, social media posts, sensor readings, videos, documents, and logs. These data types, which fall into the categories of semi-structured and unstructured data, made up the vast majority of enterprise information, yet traditional warehouses were not designed to ingest, store, or process them at scale.

The solution that emerged was the Data Lake. A Data Lake is a centralized repository that stores structured, semi-structured, and unstructured data in its raw form. Unlike a Data Warehouse, which requires data to be modeled and cleansed before it is loaded, a Data Lake follows a schema-on-read philosophy. Data is ingested quickly, without requiring upfront modeling, and structure is applied later when the data is analyzed. This makes the Data Lake highly flexible for exploratory analysis, advanced analytics, and machine learning.

Over the past decade, the terms Big Data and Data Lake have often been used interchangeably, leading to significant confusion in both technical and business conversations. While they are closely related, they are not the same. Understanding the distinction helps organizations make better architectural decisions and avoid unrealistic expectations when planning data initiatives.

Big Data: The Broader Ecosystem

Big Data is not a single technology or storage system. It is the movement and ecosystem that arose to address the challenges posed by the three Vs, volume, velocity, and variety, of modern data. Big Data includes the distributed storage and processing technologies (HDFS, MapReduce, Spark, Kafka), the new types of databases (NoSQL, graph, document, columnar), and the analytical methods that enable organizations to handle massive, fast-moving, and diverse datasets.

In short, Big Data represents the toolbox and paradigm that allowed enterprises to move beyond the limitations of traditional RDBMS and warehouses.