Knowledge Hub

Canonical Data Modelling is the practice of defining a standard, enterprise-wide representation of core data entities so that all systems within an organization can exchange information consistently. Instead of allowing each application to maintain its own version of key business objects such as Customer, Product, or Loan Application, a canonical model establishes a shared and authoritative structure that serves as the common reference point for integration. The term “canonical” refers to a standard or accepted form. In mathematics, a canonical representation expresses something in a normalized and agreed structure. In enterprise architecture, the same principle applies: a canonical model defines how a business entity should be represented when it moves between systems, even though each system may maintain its own internal structure.

In many organizations, integration evolves through point-to-point connections. In a point-to-point model, each system directly maps its data format to every other system it communicates with. While this may work in small environments, it becomes increasingly complex as more systems are added. If five systems need to exchange information, each system may require multiple custom mappings. As the enterprise grows, the number of integrations expands rapidly, creating tight coupling, duplicated transformation logic, and high maintenance overhead. Any change to one system, such as an upgrade or replacement, can require modifications in many connected systems. Over time, the integration landscape becomes fragile and difficult to manage.

JSON / XML Schema Modelling is the practice of defining structured schemas for semi-structured data formats such as JSON and XML. These schemas specify the allowed structure, elements, datatypes, and constraints of documents to ensure consistency, validation, and interoperability across systems. Rather than focusing on database storage, this modelling technique focuses on structured data exchange.

The primary goal of JSON / XML Schema Modelling is to provide a clear and enforceable contract for APIs, integration workflows, and document-based systems. When multiple applications exchange data, they must agree not only on the names of fields but also on their structure, required attributes, data types, and validation rules. A schema formalizes these expectations and prevents inconsistent or malformed data from entering the system.

In this approach, schemas define elements, nesting rules, datatypes, required fields, and sometimes value constraints. For example, a Loan schema might define that LoanID is a required string, LoanAmount is a numeric field greater than zero, and Status must be one of a predefined set of values. The schema ensures that every system sending or receiving loan data adheres to the same structural rules.

API / Service Data Modelling is the practice of designing the data contracts and structures that define how information is exchanged between services, applications, and systems. Rather than focusing on how data is stored internally, this approach concentrates on how data is exposed, requested, and consumed across interfaces.

The primary goal of API / Service Data Modelling is to ensure clarity, consistency, and stability in data exchanged across APIs and services. Well-designed API models reduce integration complexity, prevent misunderstandings between systems, and minimize runtime errors. They act as formal contracts between service providers and service consumers.

In this approach, the model defines the structure of inputs and outputs. For example, a Loan Application API may specify required request fields such as CustomerID, LoanAmount, and Term. The response may include fields such as LoanID, ApprovalStatus, and InterestRate. These definitions establish what data must be provided, what data will be returned, and under what rules. The emphasis is not on database tables but on message structures.

Framework for Big Data Processing

Lambda Architecture is an essential concept in the world of Big Data, gaining attention as technology for handling massive data volumes has advanced. It forms the backbone of many decision-support systems by providing a structured approach to data processing, integrating both batch and real-time data workflows. The Lambda Architecture model offers a comprehensive system for handling data from generation to end-user access, balancing speed and accuracy through batch and real-time layers.

The Foundations of Lambda Architecture

Lambda Architecture is grounded in lambda calculus, a formal system in mathematical logic that supports computation using function abstraction, application, and variable substitution. Lambda calculus is a universal model that can theoretically simulate any Turing machine, making it foundational to computer science and software engineering. In data processing, Lambda Architecture uses this mathematical basis to create a dual-layered system that can handle data asynchronously and in real time.

Kappa Architecture is mostly confused with Lambda Architecture which is fair as both processes Streaming as well as Batch data. Then how both are different?

Due to high demand of Real-Time data, Kappa Architecture has emerged but not for all kinds of data sizes just like Lambda Architecture is.

We have gone through in detail (in separate topic) like what is Lambda Architecture so won’t go in details like how Streaming and Batch works. Let’s understand how Kappa is different than Lambda Architecture.

Kappa Architecture offers ONLY stream processing using Queuing tools like KAFKA for both Streaming and Batch datasets. It doesn’t offer separate tools or technologies to process batches. It says, process both Real-Time Streaming and Batch data using same tool as KAFKA but the biggest differentiator here is, it stores data in a separate data store which might not be suitable to store TBs or PBs of dataset. It offers batch related downstream applications to connect directly to its storage.

Data Pipelines is as simple as traveling of data from one point to another but at the same time, the magnitude and importance of this term is as critical as, in the absence of a reliable Data Pipeline, no Decision can be taken on time, and we know ‘Time is Money’.

Data Processing Mechanisms

• Push: When source systems push data to target systems.
• Pull: When target systems pull data from source systems.
• Streaming: When data keeps flowing from source to target systems.

Data Loading Types

• Bulk Loading: This means processing the whole set of data every time from the source. We also call it ‘Data Dump’.
• CDC Loading: This means processing only changed data from the source.

Type of Data Pipeline

• Batch processing means processing data after a certain time duration e.g., start processing after 24 hours or maybe 12 hours. The better hardware is in place the small-batch processing time-duration can be.
• Real-Time processing (event base) means, the moment there is a change in the source system, that piece of data is processed towards the target e.g., the moment a post is LIKED on Facebook or LinkedIn it moves to downstream systems. To support real-time from OLTP, CDC plays a vital role as in real-time, we expect a very small size of data to travel.

Data Transformation is the bread and butter of ETL developers and data engineers.

Data Transformation is the starting point of any KPI, one needs to pull and process data based on business requirements for decision makers.

E = Extract | T = Transform | L = Load

This is the most common interview question and only those who have architecturally worked on integration can answer.

The general definitions

• ETL: Extract, Transform and Load
• ELT: Extract, Load and Transform
• ECL: Extract, Convert and Load

BUT technically HOW? Ask this nerd kind of question with experienced ETL/ ELT developers and I am sure 90% won't be able to respond.

Let’s go a little deeper and understand.

1. ETL processing is done on the source system as 1) Extract part (select) runs on source system, 2) Transform part runs on source system and 3) Load part (push data to- target table) runs on source system. Means all the operations are happening on the source system.
2. ELT processing has three more operations i.e., its DDL-EL-ETL. In ELT 1) first, a temp table is created (DDL) in the target system, 2) Extract part (select) runs on the source system, 3) Load (one-to-one push to temp table) runs on the source system, 4) Again Extract part (select) from temp table runs on the target system, 5) Transform part runs on the target system and 6) Load part (push to the table) runs on the target system. This means, 2 out of 6 operations are run on the source system where 4 operations including the most resource-hungry operation i.e., TRANSFORM operation run in the target system.
3. ECL processing is majorly used in Text Analytics where ECL tools read text from mediums like word, PDF, text file etc., catalog it by giving tags and load it in target storage which can be anything. Please note, here we are not talking about converting unstructured data like video, audio, scan, image etc., into structured data. Here, we are talking about converting or tagging existing data which is already in text format.

Business Analytics has been the most demanding cycle in the current era. Business Analytics supports DSS (Decision Support Systems) and the technologies to support started from OLTP systems to OLAP systems i.e., source systems to Data Warehouses or Data Marts or Data Lakes or Data Lake Houses or Data Vault or Business Vault or Enterprise Data Hubs or Enterprise Data Fabric etc., but that’s not the complete cycle. The cycle completes once Reverse ETL is performed.

Let’s understand with an example. For example, the product sales data is loaded in a system (OLTP) then it is pushed or pulled in a Data Lake or a Data/ Business Vault or in a Data Warehouse or Data Mart or Data Lake House etc. Once data has move in these datastores then analytics are performed using Data Science practice like by running Machine Learning, Deep Learning, and many other means to analyze data for prediction or forecasting and last but not the least the data is present to Top Management for decision marking. Management decides to initiate campaigns based on the data element extracted from the whole cycle from OLTP to OLAP etc.

Data Conversion is mostly confused with Data Transformation.

In Data Conversion, data is not changed whereas in Data Transformation, data can be changed based on business requirements for certain decision making.

• Data Conversion can be used to standardize different formats into one unified format.
• Data Conversion can be, to convert one format into different formats.
• Data Conversion can be, to convert one format to another format as per business requirement.
• Data Conversion can be, to convert one data model to another using Data Transformation process for business ease of use.
• Data Conversion can be, to convert data from data to Reporting and Dashboards formats.

Example of basic Data Conversion

• Convert weather data like from Celsius to Fahrenheit and vice versa.
• Conversion of all date formats in one format.
• Conversion of unstructured like image, pdf, video, audio etc. into semi or structured file formats
• If a field has numbers only and the data type is string, then better to convert it to number as processing is much faster on number data type as compared to string.
• If a field has predefined data like Male/ Female, Yes/ No etc. and its field data type is varchar or varchar 2 then better to convert it to char data type as processing is much faster on char data type as compared to varchar or varchar 2.

Data Parsing and Formatting sometime works interchangeably and mixed with Data Conversion and Data Transformation which is not correct.

Parsing term comes from Latin part (orationis), meaning part (of speech) [Wikipedia].

We all must have heard about parser. Yes, Data Parsing is about rearranging or formatting the data in such a way that it’s understandable or usable. In Data Parsing, data is not anonymized neither does it changes its meaning rather it changes for the sake of its understandable.

Referring to the image, we can see data in columns are rearranged and now it can make sense. If we pull it in a reports or dashboard, the decisions can be taken, of course not on un-parsed data.

In the era of social media, most of online portals gives APIs to allow third parties to extract data in a readable format but those who doesn’t provide APIs, we use Web Scrapping (explained in separate topic) to crawl the websites, online portals etc., to extract details by running different type of parsing approaches.