Knowledge Hub

HTAP stands for Hybrid Transactional/ Analytical Processing. It is a new architecture proposed by Garner in 2014.

In previous topics, we have established that there is an OLTP database for operational tasks and then Data Warehouse, Data Lake, Delta Lake, Data Lakehouse, Data/ Business Vault etc. for analysis and analysis. All keep data separately in their own storage for management decisions.

HTAP is proposing to use one system for both transaction and analytical workload. It is offering an in-memory plus columnar approach for analytical processes.

With the advancements in hardware, in-memory and Cloud, Gartner thinks it is very much possible to have both Transactional and Analytical processing on the same database.

Advantages

• No duplication of data as data will not move from the same one operational database.
• Business or Management can make decisions on actual data in real-time, in contrast if data must move from OLTP to OLAP systems.

As per the name depiction, Data Locality looks like knowing the location of the data; it is partially relevant.

Traditionally, to use data for analysis, at the backend the data is fetched from storage/ data nodes into master nodes, for computation. There is a complete history of how initially whole data was pulled then blocks were pulled then the columnar pull concept came-in, etc. so we won't go into detail about how technology has evolved to process data.

Since semi and unstructured data has become a necessary part of the Data World, it comes with huge storage requirements i.e., in TB and even in PB. To host this humongous data, clusters of 100s of storage/ data nodes are set up, so it is not possible to pull this size of data for computation in master nodes which are normally few. It was a critical requirement to process queries where data is residing.

With the introduction of Big Data/ Hadoop/ HDFS where data is stored in the shape of blocks, the concept surfaced i.e., instead of fetching data for computation, let's take the query to the storage/ data nodes, process it there and fetch only processed data. This process of moving the computation close to where the actual data resides on the storage/ data node, instead of moving large data to computation, is called Data Locality.

Query Optimization means to find the fastest route to execute the query. During this process, multiple execution plans are generated and at the end the best one is executed.

Steps involved in this process.

• Query
• Parse & Translator
• Relational Algebra
• Optimizer
• Execution Plan
• Evaluation Engine
• Query Output

The core capability of Query Optimization is to search for an execution plan which can find the least time required for query evaluation.

Phases of Query Processing

• Parsing & Translation
• Optimization
• Evaluation

Example:

We normally submit queries in SQL format. In the first phase of Parsing & Translation, SQL query is translated in an internal form where the parser validates the syntax of the query and the relations of the query in the database and so on. Then it is converted into a relational algebra expression, something like below.

Query:
Select col1 from table1 where col1=’XYZ’

Relational Algebra Expression:

Sharding

We know what Partitioning is. Sharding has similar features with some architectural differences. It also divides tables into multiple smaller datasets but in contrast with Partitions, Sharding stores each Shard on one dedicated node, so it’s called Shared-nothing Architecture. Advantages of Sharding is, referring to the same example from the topic on Partitioning, when the query runs for Year 2 & 3, each node with each Shard will process each dataset individually so will be much faster as compared to standard Partitioning where many partitions can reside on one node in Shared Architecture.

Sharding Architectures

• Key Based: It is also known as hash based Sharding, which involves using a value taken from newly written data and plugging it into a hash function to determine which shard the data should go to.
• Range Based: It involves Sharding data based on ranges of a given value.
• Directory Based: One must create and maintain a lookup table that uses a shard key to keep track of which shard holds which data.

DLOps is like MLOps and belongs to Data Science domain.

Before we try to understand DLOps as compared to MLOps, one must understand the difference between Machine Learning and Deep Learning. (There is separate topic on difference between Machine Learning and Deep Learning)

‘It won’t be wrong to say, MACHINE LEARNING is for simpler and more structured & semi-structured use cases whereas DEEP LEARNING is for complex and unstructured use cases.’

Let’s try to understand with a simple example. If we need to find out how many cars, bikes or buses there are in structured or semi-structured datasets then it’s relatively easy as data is in rows and columns format. We can have the width, length, and height of each car, bike or bus so using MACHINE LEARNING models we can easily ready this dataset and produce expected output. But what if we give pictures to ML models to extract the same result, will it be possible? The answer is No as ML models normally work on structured or semi-structured datasets. To find out cars, bikes, or buses from pictures, DEEP LEARNING models are used which can recognize the objects in the pictures and find out which presents car or bike or a bus.

There are four types of instructions to pull data i.e., SISD, SIMD, MISD and MIMD.

• SISD: Single Instruction Stream and Single Data Stream
• SIMD: Single Instruction Stream and Multiple Data Stream
• MISD: Multiple Instruction Stream and Single Data Stream
• MIMD: Multiple Instruction Stream and Multiple Data Stream

Before we jump into understanding all above mentioned techniques, let’s first understand what is a Core in a machine or in a computer or in a server etc.?

A CPU Core is built of billions of microscopic transistors to process instructions passed to it. One instruction is handled by One core, if someone needs multitasking or multiple instruction in parallel, minimum 2 cores are required.

We know when we insert or load data in a computer or server or in a node, it gets saved in hard disk. We won’t go in to details how it is stored on the hard disk. We can imagine when we write something on paper, consider the same way, data is saved in a hard disk.

Data Skew Issue is one of the main hidden culprits for query performance 😊. It is normally referred to as Skewed data distribution which is used in Data Science domain.

Have you ever noticed, your query is correct, your joins are correct, your all indexes are enabled, you have high data quality, you have data standardization in place but still your query performance is not as expected? You noticed when you loaded new datasets, queries were flying but as time passed, the performance has gone down. You have added more CPU, more RAM, more nodes but NO performance is not like newly added data. It’s because of the Data Skew Issue.

Data Skew Issue is about when data is not distribution evenly. Uneven distribution of data at cluster level causes this issue. It heavily degrades the performance of the queries especially those with joins.

We all have faced a situation where query run exceptionally good till 99% and then hangs for 3-4 hours right lolz yeah, I can understand your pain 😊. Let me tell you the reason, it’s because of Data Skew Issue.

A Watermark is a recognizable image or pattern embedded into a digital or physical document to identify its authenticity or to protect the document from unauthorized use. Watermarks are commonly used in various contexts, including photography, art, documents, and currency.

In the context of digital media, such as images or videos, a watermark is often a semi-transparent logo or text that is overlaid onto the content. This watermark serves several purposes:

• Copyright Protection: Content creators often use watermarks to assert their ownership of the material. This discourages unauthorized use and helps in identifying the original creator.
• Branding: Watermarks can also be used for branding purposes. Companies may add their logos or brand names as watermarks to promote brand recognition even when the content is shared online.
• Security: Watermarks can be used to secure confidential or sensitive documents. For example, companies might add watermarks to internal documents to discourage leaks or unauthorized distribution.
• Authentication: In the case of physical documents, such as banknotes or certificates, watermarks can be embedded during the papermaking process. These watermarks serve as a form of authentication to verify the document's legitimacy.

A Unified Approach to Data Ecosystem Management

The concept of Data Platform Gravity revolves around a holistic framework where every aspect of data handling, from ingestion to transformation, analytics, and deployment, is achieved within a single, unified platform. Inspired by the Data Fabric paradigm, it advocates for seamless operations across the entire data lifecycle, reducing complexities, minimizing data silos, and empowering organizations to derive value efficiently and effectively.

Understanding Data Platform Gravity

In a traditional data management ecosystem, disparate tools and technologies often create bottlenecks. A single project might involve different platforms for:

• Extracting and loading data from sources,
• Transforming and enriching data,
• Performing analytics and data science experiments,
• Visualizing results in dashboards or reports, and
• Sharing data insights with external systems or stakeholders.

An Execution Graph is a visual and logical representation of how tasks, operations, or computations are executed in a system. It shows the order in which activities run and how they depend on one another.

In data engineering and distributed computing, an execution graph is often represented as a Directed Acyclic Graph (DAG). “Directed” means tasks move in one direction. “Acyclic” means the graph does not loop back on itself. Once a task finishes, the flow moves forward.

In simple terms, an execution graph answers one important question:

What runs first, what runs next, and what depends on what?

Why Execution Graphs Matter

Modern data systems rarely execute tasks in a straight line. Instead, they perform multiple steps such as reading data, filtering, transforming, aggregating, and storing results. Some steps can run in parallel, while others must wait for previous tasks to complete.

An execution graph helps systems: