Knowledge Hub

Immutable Data Warehouse has come from immutable data. Data folks are aware of it but don’t use it more often.

‘Immutable means not changeable.’

Now, the question is how it’s relevant to us in Data World. We know when we are inserting new data in a database, at the same time we also have the option to change it by using update or delete commands which means data is mutable.

Mutable feature is a MUST TO HAVE in any database at a technology level, means, any databases or data store MUST allow to update or delete. One should or should not update or delete, can be a business requirement but cannot be technology limitation.

‘By default, Data Warehouse is an Immutable datastore.’

Immutable feature is not only about, not able to update or delete data, it’s about which data should be immutable, and which should be mutable. For this, one must understand the data.

For example, CUSTOMER dataset contains many types of data e.g., name, date of birth, parents, children, address, contact details, jobs, cars etc. In this dataset, you must have judged which information is immutable and which mutable.

A Logical Data Warehouse (LDW) is not a new storage technology but an architectural approach to delivering a unified data warehouse view without physically consolidating all data into one centralized repository. Unlike a traditional physical data warehouse, which extracts, transforms, and loads data into a dedicated storage layer, a Logical Data Warehouse integrates data virtually. It connects distributed data sources and presents them as a single, consistent analytical environment.

The concept does not replace traditional modeling approaches. A Logical Data Warehouse can still support third normal form structures aligned with the philosophy of Bill Inmon or dimensional modeling approaches such as star and snowflake schemas aligned with Ralph Kimball. The distinction lies not in the modeling technique but in the physical location of the data. In an LDW, data may remain in source systems, data lakes, physical warehouses, business vault layers, delta-based platforms, streaming systems, or APIs. The integration happens logically rather than physically.

Data Mesh is normally confused with Data Mashup (explained separately), but both are totally different.

Data Mesh is a modern approach to data architecture and governance that emphasizes decentralization and domain-driven ownership of data products. Unlike traditional centralized data systems, which rely on a monolithic team to manage and deliver data assets across an organization, Data Mesh shifts the focus towards empowering individual domains to become autonomous in managing their data products. This paradigm fosters scalability, improves data usability, and ensures alignment with specific business needs.

Understanding the Foundations of Data Mesh

At its core, Data Mesh revolves around four key principles:

1. Domain-Oriented Decentralization: The fundamental premise of Data Mesh is the distribution of data ownership to individual domains. A domain, in this context, refers to a specific business area or operational unit, such as marketing, sales, or customer service. Each domain is responsible for defining, managing, and maintaining its own data products. By doing so, Data Mesh reduces the bottlenecks and inefficiencies associated with centralized data teams.
2. Data as a Product (DAaP): Data Mesh introduces the concept of treating data as a product. This means that data products must be designed with the same rigor and customer-centricity as any software product. Each data product must:

HTAP stands for Hybrid Transactional/ Analytical Processing. It is a new architecture proposed by Garner in 2014.

In previous topics, we have established that there is an OLTP database for operational tasks and then Data Warehouse, Data Lake, Delta Lake, Data Lakehouse, Data/ Business Vault etc. for analysis and analysis. All keep data separately in their own storage for management decisions.

HTAP is proposing to use one system for both transaction and analytical workload. It is offering an in-memory plus columnar approach for analytical processes.

With the advancements in hardware, in-memory and Cloud, Gartner thinks it is very much possible to have both Transactional and Analytical processing on the same database.

Advantages

• No duplication of data as data will not move from the same one operational database.
• Business or Management can make decisions on actual data in real-time, in contrast if data must move from OLTP to OLAP systems.

As per the name depiction, Data Locality looks like knowing the location of the data; it is partially relevant.

Traditionally, to use data for analysis, at the backend the data is fetched from storage/ data nodes into master nodes, for computation. There is a complete history of how initially whole data was pulled then blocks were pulled then the columnar pull concept came-in, etc. so we won't go into detail about how technology has evolved to process data.

Since semi and unstructured data has become a necessary part of the Data World, it comes with huge storage requirements i.e., in TB and even in PB. To host this humongous data, clusters of 100s of storage/ data nodes are set up, so it is not possible to pull this size of data for computation in master nodes which are normally few. It was a critical requirement to process queries where data is residing.

With the introduction of Big Data/ Hadoop/ HDFS where data is stored in the shape of blocks, the concept surfaced i.e., instead of fetching data for computation, let's take the query to the storage/ data nodes, process it there and fetch only processed data. This process of moving the computation close to where the actual data resides on the storage/ data node, instead of moving large data to computation, is called Data Locality.

Query Optimization means to find the fastest route to execute the query. During this process, multiple execution plans are generated and at the end the best one is executed.

Steps involved in this process.

• Query
• Parse & Translator
• Relational Algebra
• Optimizer
• Execution Plan
• Evaluation Engine
• Query Output

The core capability of Query Optimization is to search for an execution plan which can find the least time required for query evaluation.

Phases of Query Processing

• Parsing & Translation
• Optimization
• Evaluation

Example:

We normally submit queries in SQL format. In the first phase of Parsing & Translation, SQL query is translated in an internal form where the parser validates the syntax of the query and the relations of the query in the database and so on. Then it is converted into a relational algebra expression, something like below.

Query:
Select col1 from table1 where col1=’XYZ’

Relational Algebra Expression:

Sharding

We know what Partitioning is. Sharding has similar features with some architectural differences. It also divides tables into multiple smaller datasets but in contrast with Partitions, Sharding stores each Shard on one dedicated node, so it’s called Shared-nothing Architecture. Advantages of Sharding is, referring to the same example from the topic on Partitioning, when the query runs for Year 2 & 3, each node with each Shard will process each dataset individually so will be much faster as compared to standard Partitioning where many partitions can reside on one node in Shared Architecture.

Sharding Architectures

• Key Based: It is also known as hash based Sharding, which involves using a value taken from newly written data and plugging it into a hash function to determine which shard the data should go to.
• Range Based: It involves Sharding data based on ranges of a given value.
• Directory Based: One must create and maintain a lookup table that uses a shard key to keep track of which shard holds which data.

DLOps is like MLOps and belongs to Data Science domain.

Before we try to understand DLOps as compared to MLOps, one must understand the difference between Machine Learning and Deep Learning. (There is separate topic on difference between Machine Learning and Deep Learning)

‘It won’t be wrong to say, MACHINE LEARNING is for simpler and more structured & semi-structured use cases whereas DEEP LEARNING is for complex and unstructured use cases.’

Let’s try to understand with a simple example. If we need to find out how many cars, bikes or buses there are in structured or semi-structured datasets then it’s relatively easy as data is in rows and columns format. We can have the width, length, and height of each car, bike or bus so using MACHINE LEARNING models we can easily ready this dataset and produce expected output. But what if we give pictures to ML models to extract the same result, will it be possible? The answer is No as ML models normally work on structured or semi-structured datasets. To find out cars, bikes, or buses from pictures, DEEP LEARNING models are used which can recognize the objects in the pictures and find out which presents car or bike or a bus.

We touched base with Data Mining and Data Science, it will be ideal to discuss Data Dredging, Snooping, p-hacking, and Fishing. New terms right 😊?

Data Dredging, Snooping and Fishing all refer to the same behavior of data analysis BUT without proper hypothesis and relationship among datasets.

Data mining finds results based on the correlation of data in large data sets, but Data Dredging, Snooping, p-hacking, and Fishing find results based on chance methodology. In other words, Data Dredging, Snooping, p-hacking, and Fishing share the results which require more investigation. These sometimes bypass Data Mining techniques and come up with immature conclusions. On occasions, it shows more details about something than it contains.

There are two types of Fact Findings in any analysis that ultimately assists in DSS i.e., 1) known facts 2) unknown facts. Data mining techniques extract Known Facts as it’s done only on structured datasets and mostly, we know what's there in structured data like relationships between tables and relationships of data within the same table. But that's Not True, sometimes, even in structured data there are Unknown Facts and that’s where Data Dredging, Snooping and Fishing techniques come into the action.

There are four types of instructions to pull data i.e., SISD, SIMD, MISD and MIMD.

• SISD: Single Instruction Stream and Single Data Stream
• SIMD: Single Instruction Stream and Multiple Data Stream
• MISD: Multiple Instruction Stream and Single Data Stream
• MIMD: Multiple Instruction Stream and Multiple Data Stream

Before we jump into understanding all above mentioned techniques, let’s first understand what is a Core in a machine or in a computer or in a server etc.?

A CPU Core is built of billions of microscopic transistors to process instructions passed to it. One instruction is handled by One core, if someone needs multitasking or multiple instruction in parallel, minimum 2 cores are required.

We know when we insert or load data in a computer or server or in a node, it gets saved in hard disk. We won’t go in to details how it is stored on the hard disk. We can imagine when we write something on paper, consider the same way, data is saved in a hard disk.