Knowledge Hub

Epistemology, a term rooted in the Greek word episteme, meaning “knowledge,” and logos, meaning “discourse” or “communication,” explores the concept of “justified knowledge.” It addresses questions like “What do we know?”, “How can we claim to know something?”, “What validates our beliefs?”, and “How do we ascertain the certainty of our knowledge?”

In the realm of data, these questions take on a new dimension, especially in the Big Data era, where we store vast amounts of raw data, structured, semi-structured, and unstructured. Data science teams apply machine learning or deep learning models to extract insights, turning raw data into knowledge. However, to ensure that these insights align with their origins, a “justification of knowledge” is essential, ensuring that what we extract truly reflects what was expected. This process emphasizes evidence-based validation, where qualitative research relies on the quality of design and data collected.

CDC and Real-Time, a couple of terms which people are confused with and assumes both are same but in the real world, both are different topics.

CDC: Full abbreviation of CDC is Change Data Capture, meaning the moment, the data is changed in database ONLY THE CHANGED DATA gets picked up by the CDC process and pushed to next layer. Changed data can be like 1) a new row can be added or 2) an existing row can be updated or 3) a previously added row can be deleted or dropped.

This whole process can be Real-Time but mostly can't as Real-Time + CDC needs a lot of memory processing which comes with heavy TCO, so it all depends on ROI for the related Business Case.

There are Four CDC methods:

1. Keep the previous data set in table1, get a new data set in table2. Compare both and extract rows that are deleted or are new or have changed since last time. This is the slowest method and requires the heaviest processing either on the source (not recommended) or on target system.
2. Source Tables have date columns that can identify the latest inserted or updated records. Using these columns, CDC data can be extracted easily. This is the easy approach, but it requires the Source Data Model to have date fields in each table which if not requires extensive development effort both at the backend database and frontend Application. It has a processing impact on the source system as well.
3. This method is mostly used where Transaction Logs are read. Transaction Log is generated when an insert/ update/ delete operation takes place on a table. In this method, the log is read instead of hitting the source tables so no impact on the source database but minimal impact on I/O [recommended and mostly used approach].
4. This is the most recent method and very few solutions are using it. In this method, the data is read at the application level, in other words when the Application writes in Database, at the same time it pushes the operation for CDC processing. No impact on source database or I/O.

ESP (Event-Stream Processing) is a term used along with CDC and Real-Time processing.

As we touched based on Real-Time processing in another topic, ESP is exactly what Real-Time is all about. In other words, the moment any activity has happened in source system or in source application which can be online portal, social media or a mobile etc., so the moment any activity takes place, the latest action is pushed to next stage. Next stage can be a Data Lake or Data Warehouse, or it can be intermediately solution in in-memory to prepare a response to send directly back to customer’s mobile.

‘Even-Streaming Processing is heavily used if a reaction is planned in Real-Time.’

For example, a customer buys a product from a shop in a mall, the moment he or she use their credit card for payment, along with OTP code another message comes to their mobile, telling them there is a sales or promotion going on in this mall, please have a look before making this payment. This is called Real-Time Even-Streaming Processing.

Immutable Data Warehouse has come from immutable data. Data folks are aware of it but don’t use it more often.

‘Immutable means not changeable.’

Now, the question is how it’s relevant to us in Data World. We know when we are inserting new data in a database, at the same time we also have the option to change it by using update or delete commands which means data is mutable.

Mutable feature is a MUST TO HAVE in any database at a technology level, means, any databases or data store MUST allow to update or delete. One should or should not update or delete, can be a business requirement but cannot be technology limitation.

‘By default, Data Warehouse is an Immutable datastore.’

Immutable feature is not only about, not able to update or delete data, it’s about which data should be immutable, and which should be mutable. For this, one must understand the data.

For example, CUSTOMER dataset contains many types of data e.g., name, date of birth, parents, children, address, contact details, jobs, cars etc. In this dataset, you must have judged which information is immutable and which mutable.

A Logical Data Warehouse (LDW) is not a new storage technology but an architectural approach to delivering a unified data warehouse view without physically consolidating all data into one centralized repository. Unlike a traditional physical data warehouse, which extracts, transforms, and loads data into a dedicated storage layer, a Logical Data Warehouse integrates data virtually. It connects distributed data sources and presents them as a single, consistent analytical environment.

The concept does not replace traditional modeling approaches. A Logical Data Warehouse can still support third normal form structures aligned with the philosophy of Bill Inmon or dimensional modeling approaches such as star and snowflake schemas aligned with Ralph Kimball. The distinction lies not in the modeling technique but in the physical location of the data. In an LDW, data may remain in source systems, data lakes, physical warehouses, business vault layers, delta-based platforms, streaming systems, or APIs. The integration happens logically rather than physically.

Data Mesh is normally confused with Data Mashup (explained separately), but both are totally different.

Data Mesh is a modern approach to data architecture and governance that emphasizes decentralization and domain-driven ownership of data products. Unlike traditional centralized data systems, which rely on a monolithic team to manage and deliver data assets across an organization, Data Mesh shifts the focus towards empowering individual domains to become autonomous in managing their data products. This paradigm fosters scalability, improves data usability, and ensures alignment with specific business needs.

Understanding the Foundations of Data Mesh

At its core, Data Mesh revolves around four key principles:

1. Domain-Oriented Decentralization: The fundamental premise of Data Mesh is the distribution of data ownership to individual domains. A domain, in this context, refers to a specific business area or operational unit, such as marketing, sales, or customer service. Each domain is responsible for defining, managing, and maintaining its own data products. By doing so, Data Mesh reduces the bottlenecks and inefficiencies associated with centralized data teams.
2. Data as a Product (DAaP): Data Mesh introduces the concept of treating data as a product. This means that data products must be designed with the same rigor and customer-centricity as any software product. Each data product must:

HTAP stands for Hybrid Transactional/ Analytical Processing. It is a new architecture proposed by Garner in 2014.

In previous topics, we have established that there is an OLTP database for operational tasks and then Data Warehouse, Data Lake, Delta Lake, Data Lakehouse, Data/ Business Vault etc. for analysis and analysis. All keep data separately in their own storage for management decisions.

HTAP is proposing to use one system for both transaction and analytical workload. It is offering an in-memory plus columnar approach for analytical processes.

With the advancements in hardware, in-memory and Cloud, Gartner thinks it is very much possible to have both Transactional and Analytical processing on the same database.

Advantages

• No duplication of data as data will not move from the same one operational database.
• Business or Management can make decisions on actual data in real-time, in contrast if data must move from OLTP to OLAP systems.

As per the name depiction, Data Locality looks like knowing the location of the data; it is partially relevant.

Traditionally, to use data for analysis, at the backend the data is fetched from storage/ data nodes into master nodes, for computation. There is a complete history of how initially whole data was pulled then blocks were pulled then the columnar pull concept came-in, etc. so we won't go into detail about how technology has evolved to process data.

Since semi and unstructured data has become a necessary part of the Data World, it comes with huge storage requirements i.e., in TB and even in PB. To host this humongous data, clusters of 100s of storage/ data nodes are set up, so it is not possible to pull this size of data for computation in master nodes which are normally few. It was a critical requirement to process queries where data is residing.

With the introduction of Big Data/ Hadoop/ HDFS where data is stored in the shape of blocks, the concept surfaced i.e., instead of fetching data for computation, let's take the query to the storage/ data nodes, process it there and fetch only processed data. This process of moving the computation close to where the actual data resides on the storage/ data node, instead of moving large data to computation, is called Data Locality.

Query Optimization means to find the fastest route to execute the query. During this process, multiple execution plans are generated and at the end the best one is executed.

Steps involved in this process.

• Query
• Parse & Translator
• Relational Algebra
• Optimizer
• Execution Plan
• Evaluation Engine
• Query Output

The core capability of Query Optimization is to search for an execution plan which can find the least time required for query evaluation.

Phases of Query Processing

• Parsing & Translation
• Optimization
• Evaluation

Example:

We normally submit queries in SQL format. In the first phase of Parsing & Translation, SQL query is translated in an internal form where the parser validates the syntax of the query and the relations of the query in the database and so on. Then it is converted into a relational algebra expression, something like below.

Query:
Select col1 from table1 where col1=’XYZ’

Relational Algebra Expression:

Sharding

We know what Partitioning is. Sharding has similar features with some architectural differences. It also divides tables into multiple smaller datasets but in contrast with Partitions, Sharding stores each Shard on one dedicated node, so it’s called Shared-nothing Architecture. Advantages of Sharding is, referring to the same example from the topic on Partitioning, when the query runs for Year 2 & 3, each node with each Shard will process each dataset individually so will be much faster as compared to standard Partitioning where many partitions can reside on one node in Shared Architecture.

Sharding Architectures

• Key Based: It is also known as hash based Sharding, which involves using a value taken from newly written data and plugging it into a hash function to determine which shard the data should go to.
• Range Based: It involves Sharding data based on ranges of a given value.
• Directory Based: One must create and maintain a lookup table that uses a shard key to keep track of which shard holds which data.