Knowledge Hub

Event-Driven Data Modelling structures data around events, discrete occurrences that capture something that has happened, typically with a timestamp and contextual details. Instead of centering the model on static entities such as Customer or Loan, this approach treats events as the primary unit of design. Each event represents a fact in time.

The goal of event-driven modelling is to enable real-time processing, streaming architectures, and event sourcing patterns. By designing systems around events, organizations can react immediately to business activity while maintaining a complete, replayable history of what occurred.

In this model, the focus shifts from storing the current state of entities to recording event streams. Events are immutable and append-only. Once an event is written, it is never updated or deleted. Typical events in a loan approval domain might include LoanApplicationSubmitted, LoanApplicationApproved, or RepaymentMade. Each event contains identifying information, contextual data, and a timestamp.

CDC and Real-Time, a couple of terms which people are confused with and assumes both are same but in the real world, both are different topics.

CDC: Full abbreviation of CDC is Change Data Capture, meaning the moment, the data is changed in database ONLY THE CHANGED DATA gets picked up by the CDC process and pushed to next layer. Changed data can be like 1) a new row can be added or 2) an existing row can be updated or 3) a previously added row can be deleted or dropped.

This whole process can be Real-Time but mostly can't as Real-Time + CDC needs a lot of memory processing which comes with heavy TCO, so it all depends on ROI for the related Business Case.

There are Four CDC methods:

1. Keep the previous data set in table1, get a new data set in table2. Compare both and extract rows that are deleted or are new or have changed since last time. This is the slowest method and requires the heaviest processing either on the source (not recommended) or on target system.
2. Source Tables have date columns that can identify the latest inserted or updated records. Using these columns, CDC data can be extracted easily. This is the easy approach, but it requires the Source Data Model to have date fields in each table which if not requires extensive development effort both at the backend database and frontend Application. It has a processing impact on the source system as well.
3. This method is mostly used where Transaction Logs are read. Transaction Log is generated when an insert/ update/ delete operation takes place on a table. In this method, the log is read instead of hitting the source tables so no impact on the source database but minimal impact on I/O [recommended and mostly used approach].
4. This is the most recent method and very few solutions are using it. In this method, the data is read at the application level, in other words when the Application writes in Database, at the same time it pushes the operation for CDC processing. No impact on source database or I/O.

ESP (Event-Stream Processing) is a term used along with CDC and Real-Time processing.

As we touched based on Real-Time processing in another topic, ESP is exactly what Real-Time is all about. In other words, the moment any activity has happened in source system or in source application which can be online portal, social media or a mobile etc., so the moment any activity takes place, the latest action is pushed to next stage. Next stage can be a Data Lake or Data Warehouse, or it can be intermediately solution in in-memory to prepare a response to send directly back to customer’s mobile.

‘Even-Streaming Processing is heavily used if a reaction is planned in Real-Time.’

For example, a customer buys a product from a shop in a mall, the moment he or she use their credit card for payment, along with OTP code another message comes to their mobile, telling them there is a sales or promotion going on in this mall, please have a look before making this payment. This is called Real-Time Even-Streaming Processing.

A Logical Data Warehouse (LDW) is not a new storage technology but an architectural approach to delivering a unified data warehouse view without physically consolidating all data into one centralized repository. Unlike a traditional physical data warehouse, which extracts, transforms, and loads data into a dedicated storage layer, a Logical Data Warehouse integrates data virtually. It connects distributed data sources and presents them as a single, consistent analytical environment.

The concept does not replace traditional modeling approaches. A Logical Data Warehouse can still support third normal form structures aligned with the philosophy of Bill Inmon or dimensional modeling approaches such as star and snowflake schemas aligned with Ralph Kimball. The distinction lies not in the modeling technique but in the physical location of the data. In an LDW, data may remain in source systems, data lakes, physical warehouses, business vault layers, delta-based platforms, streaming systems, or APIs. The integration happens logically rather than physically.

A Watermark is a recognizable image or pattern embedded into a digital or physical document to identify its authenticity or to protect the document from unauthorized use. Watermarks are commonly used in various contexts, including photography, art, documents, and currency.

In the context of digital media, such as images or videos, a watermark is often a semi-transparent logo or text that is overlaid onto the content. This watermark serves several purposes:

• Copyright Protection: Content creators often use watermarks to assert their ownership of the material. This discourages unauthorized use and helps in identifying the original creator.
• Branding: Watermarks can also be used for branding purposes. Companies may add their logos or brand names as watermarks to promote brand recognition even when the content is shared online.
• Security: Watermarks can be used to secure confidential or sensitive documents. For example, companies might add watermarks to internal documents to discourage leaks or unauthorized distribution.
• Authentication: In the case of physical documents, such as banknotes or certificates, watermarks can be embedded during the papermaking process. These watermarks serve as a form of authentication to verify the document's legitimacy.

The role of ontology in data integration and harmonization is to describe structure and relations. Ontologies enable a coherent structure for data across systems, functions, and domains. As the cost of maintaining and point-to-point integrations is growing along with the cost for running specific data pipelines there is a need for a new approach. By having an ontological approach to an organization’s data-driven strategy cost control will be easier to achieve.

The Concept of Ontology in Data Management

Ontologies in data management involve the use of a shared vocabulary and a set of relationships and rules to model a domain. They provide a common language for data and thus enable different systems and stakeholders to 'speak' the same language.

Data Integration Challenges

• Disparate Data Sources: Organizations often deal with data from various sources each with their unique format and structure.
• Semantic Discrepancies: Different interpretations of data across systems can lead to inconsistencies and misunderstandings.

The worlds of structured data and textual data are as far apart as the ocean is from the desert. These worlds are extremely different. But there is great value in being able to bring them together.

To that end there is textual ETL. Textual ETL reads text and disambiguates the text. As data is placed into a structured format from a textual format, the word that has been found and the context of the word that has been found are captured. That is the mechanics of how textual ETL operates.

How does textual ETL know what words to pluck out from text? Textual ETL knows by fiat of one or more taxonomies. The taxonomies alert textual ETL as to what words are important and need to be placed into a database for the purpose of analysis. In addition, the taxonomies provide the context for the word that has been selected.

Another feature of textual is inline contextualization. While many words can be contextualized by the usage of taxonomies, not all words can be treated this way. In addition to taxonomies, inline contextualization allows the word and its context to be selected by means of the positioning of the word in raw text and the proximity of the word or phrase to other words and phrases.

An Execution Graph is a visual and logical representation of how tasks, operations, or computations are executed in a system. It shows the order in which activities run and how they depend on one another.

In data engineering and distributed computing, an execution graph is often represented as a Directed Acyclic Graph (DAG). “Directed” means tasks move in one direction. “Acyclic” means the graph does not loop back on itself. Once a task finishes, the flow moves forward.

In simple terms, an execution graph answers one important question:

What runs first, what runs next, and what depends on what?

Why Execution Graphs Matter

Modern data systems rarely execute tasks in a straight line. Instead, they perform multiple steps such as reading data, filtering, transforming, aggregating, and storing results. Some steps can run in parallel, while others must wait for previous tasks to complete.

An execution graph helps systems: