Knowledge Hub

Data Streaming is a method of processing and transmitting data continuously and in real-time rather than in batches. It involves the continuous flow of data from sources to destinations, allowing for the rapid analysis, processing, and dissemination of information.

To understand Data Streaming, suppose data streaming is like water moving in a river on regular basis as compared to a place where water is stored one time with no more coming in. In traditional methods, we collect a bunch of data, like filling a bucket, and then analyze it all at once. On the other hand, in data streaming, it’s like regular pieces of data keep coming in piece by piece, in smaller size, rather than we pull in a huge size in one-go.

Few examples for better understanding.

• Social Media Feeds: Social media platforms use data streaming to show new posts, comments, and updates in real-time. For example, when you refresh your Instagram feed, you instantly see new photos and comments as they are posted.
• Weather Monitoring: Meteorologists use data streaming to receive and analyze continuous weather data. For example, temperature, humidity, and wind speed information are continuously sent and analyzed to provide real-time weather updates.
• Financial Transactions: Banks use data streaming to process and monitor financial transactions in real-time. For example, when you make a purchase with your credit card, the transaction is immediately reflected in your account balance.
• Health Monitoring Devices: Wearable devices use data streaming to provide real-time health information. For example, a fitness tracker continuously measures your heart rate, steps, and sleep patterns, updating your health stats in real-time.
• Traffic Monitoring: Traffic control systems use data streaming to monitor and manage traffic flow. For example, cameras and sensors on roads continuously provide real-time information about traffic conditions, helping optimize traffic signals.
• Online Video Streaming: Services like YouTube or Netflix use data streaming to deliver videos in real-time. For example, when you watch a video online, it starts playing almost instantly, with the data being streamed to your device as you watch.

First, I wanted to name this topic as Data or Delta Sharing but to make it more interesting, kept it as Delta Sharing ONLY 😊.

Let’s decode it…..

We all have heard about Data Sharing. It is about sharing data across departments, across branches, across organizations or in DaaS model (explained separately) with the open wide world. In preceding era, Data was not huge, Data was not treated as THE SUCCESS factor, but it felt it is. So, sharing data to other entities was OK, giving them data itself was OK.

But in the current era where data has exponentially grown, and data has become THE SUCCESS factor for any organization. Plus, due to the data size, it’s no longer possible to transfer data even if it is, it’s not in timely manner, secondly due to Data Sovereignty, organizations don’t want their data to move out of their datastores.

Above scenario was even possible in previous era where, to others, we used to give access in our environment but on a very limited basis as accessing our environment or our databases used to impact our performance and used to impact our applications which was not acceptable, that is the reason we started masking or anonymizing data and started exporting actual data to other datastores.

CDC framework is much more than CDC as a feature or as a capability by a tool. There are many tools in the market which can just extract CDC data from database which is only the first step in whole CDC framework.

CDC: Full abbreviation of CDC is Change Data Capture, meaning the moment, the data is changed in database ONLY THE CHANGED DATA gets picked up by the CDC process and pushed to next layer. Changed data can be like 1) a new row can be added or 2) an existing row can be updated or 3) a previously added row can be deleted or dropped.

This whole process can be Real-Time but mostly can't as Real-Time + CDC needs a lot of memory processing which comes with heavy TCO, so it all depends on ROI for the related Business Case.

CDC Framework: Extracting only changed data is not the end of settling data to be presented in Dashboards or Reporting, there are two more steps one must complete before changed data is considered useful.

These are one of the most confusing topics. We all mix-up data quality fixes and standardization of data for better processing with transforming data for better processing and serving.

Let’s decode it…..

We all know, in Data Warehouse or when we pull data from an OLTP system, we first store it in a LANDING ZONE somewhere which normally is the true copy of source systems. Then we move it to STAGING ZONE before we transform it into a Unified Data Warehouse (Bill Inmon way) or in a Dimensional Model (Ralph Kimble way).

Tables in LANDING and STANGING ZONES, both keep the true DATA MODEL from source systems but not the CONDITION of DATA.

WHAT DOES ABOVE STATEMENT MEANS, not the CONDITION of DATA?

Data Standardization

Does standardizing the condition of data not equal to transformation? NOPS, ITS NOT. Data Standardization can have unlimited scenarios, few are shared below e.g.,

• Standardization of Dates:
  • Converting different dates formats from different source systems e.g., DD/MM/YY, MM/DD/YY, YY/MM/DD, YY/DD/MM etc., into one format e.g., DD/MM/YYYY.
  • Convert empty date rows into e.g., 01/01/9999.
• Convert special characters to NA.
• Convert empty fields to NA.
• Profile the fields, if data type is string or varchar but data is only number than convert it into digit or number as running queries on digit or numbers are way faster.
• If the field contains codes only with only character, then make it CHAR and if it only contains numbers then make it DIGIT.
• If the full name of a person is in one field, convert it in First, Middle and Last Name.
• Fix the spelling mistakes.
• Fix the up and lower cases e.g., if a country is Australia and in data its coming as Aus, Aust, australia, Aussie, aus etc., than convert it in one format e.g., Australia.

Semantics are different for everyone. In an organization, each department has its own semantics, to view data. Same data can have different meanings for one department as compared to others.

Salary for HR department is to know who is getting what and when they hire new candidate, they can refer to what others are getting based on role, JD, skill set etc.. But Salary for Finance department can be, to calculate annual expense for the company.

Semantic Layer in Decision Support Systems are based on business requirements, in their language, using their own key words, their codes etc.

Interoperability on the other hand is about exchanging something across systems so others don’t need to regenerate it again and again.

Semantic Interoperability is about exchanging semantic information across systems or across departments or across organizations. But this comes with a challenge whether the receiving party can understand the keywords, symbols, terminologies etc., from the sending party.

ETLT is a way of talking about how we handle information. Think of it like this: first, we take out the data we need. Then, we change it to make sure it's clean and ready to use. After that, we put it where we want it to be. Lastly, we might make some more changes to the data to make it even better for understanding. ETLT is like a helpful addition to the usual process (ETL) that makes our data even more useful and organized.

ETLT stands for "Extract, Transform, Load, and Transform" in the context of data integration and processing. Please don’t worry about where this ETLT has come from. ETLT just includes another stage in ETL world to support semantic layers.

Let’s understand each:

• Extract: Involves extracting data from various sources, which could be databases, applications, logs, or other systems.
• Transform (First T): Refers to the process of cleaning, structuring, and enriching the extracted data so that it is suitable for analysis. Transformation may include data cleansing, normalization, aggregation, and other operations.
• Load: Involves loading the transformed data into a target database or data warehouse where it can be used for reporting, analysis, or other purposes.
• Transform (Second T): This additional transformation step might be included in some interpretations of ETLT. It represents further processing or transformation that occurs after the data has been loaded into the target system. This step acknowledges that data transformation can be an ongoing and iterative process.

In the world of data, synchronization and propagation are like superheroes that make sure our information stays accurate and consistent, even when it's spread out in different places.

Let's understand these ideas:

Synchronization:

• Synchronization is about keeping data the same and up to date in many different spots or systems.
• The goal is to ensure that if we change something in one place, like updating the inventory in an online store, those changes quickly show up everywhere else, such as on the website and in the physical store's system. This way, everyone using the data gets the latest and correct info.
• For example, imagine you run an online store, and you add new items to your inventory on your computer. Synchronization is like magic that makes sure those new items appear on your website and in your store's computer, so customers and the store both know what's available.

Propagation:

• Propagation is like the spreading of news, but for changes in data. It's about making sure updates in one part of a system travel to other connected parts.
• When something changes, like depositing money at an ATM in a bank, propagation helps that change reach all the places that need to know, such as your online bank account and the main bank database. It's like making sure the news of the deposit goes to all the right places.
• For example, if you put money into an ATM, propagation ensures that the information about your deposit reaches all the important places, like your online bank app and the bank's main computer. It's like making sure everyone in the banking system knows about the deposit.

In the data world, when we work with databases, we often encounter scenarios where the data we want to query and analyze resides outside the database itself. This could be in the form of files, such as CSV (Comma-Separated Values), Parquet, or JSON files, stored in external storage systems like Hadoop Distributed File System (HDFS) or cloud storage platforms such as Amazon S3 or Azure Blob Storage.

Now, imagine a situation where we have large datasets stored in external files, and we want to treat them like database tables for analytical purposes. Here comes the concept of external tables.

External Tables are a database feature that allows you to define a table structure within the database, but the actual data is stored externally, typically in files. To create an external table, you specify the table structure (columns, data types) in the database, and you also provide a location that points to the external storage where the data files are stored.

Yes, you are right, in Data World, Staging Layer is one of the most important zones then why to write about it. Trust me it's been underutilized in most implementations.

Let's decode it.....

The Staging Layer in data architecture plays a crucial role in the data processing pipeline, providing a temporary and structured area for incoming data before it is integrated into the data warehouse or other storage systems.

Many a times Staging Layer is confused with Landing Layer. Yeah trust me, I have seen people doing nothing in Staging Layer except minor things. Please note, especially in the era of cloud, if one doesn’t properly utilize Staging Layer, the consumption cost from Staging Layer to Transformed Layer will be significantly higher.

The importance of the staging layer lies in several key aspects:

• Data Quality and Validation: The staging layer acts as a checkpoint for data quality and validation. Before data is moved to the main storage or processing area, it undergoes validation checks to ensure it meets quality standards. This helps identify and address issues such as missing values, formatting errors, or inconsistencies early in the data pipeline.
• Data Enrichment: Staging allows for data enrichment processes to take place before the data reaches its final destination. This step involves cleaning, enhancing the data, making it more suitable for analysis and reporting. It ensures that the data conforms to the desired format and structure.
• Incremental Loading: The staging layer facilitates incremental loading of data, allowing for efficient updates and additions. Instead of reloading the entire dataset, only new or modified data needs to be processed, reducing the computational load and improving overall system performance.
• Data Auditing and Lineage: Staging provides a historical record of the ingested data, enabling auditing and tracing of changes over time. This is essential for tracking the lineage of data, understanding its origin, and ensuring compliance with data governance and regulatory requirements.
• Fault Tolerance and Recovery: Staging serves as a buffer against data processing failures. If an error occurs during data integration or transformation, the staging layer allows for easier recovery without affecting the main data storage. It provides a level of fault tolerance and helps maintain the integrity of the overall data processing pipeline.
• Isolation of Source and Destination Systems: Staging separates the source systems (where data originates) from the destination systems (where data is stored for analysis). This isolation ensures that any changes or issues in the source systems do not immediately impact the data warehouse or analytics environment, providing a layer of protection.
• Performance Optimization: By pre-processing and validating data in the staging layer, the main data storage systems are relieved of some of the computational overhead. This optimization contributes to improved performance and responsiveness in data processing.

EDI, or Electronic Data Interchange has been introduced since the seventies. The main use case addressed by EDI is to automate data exchange between trading partners (Customers and Suppliers) for business documents, such as purchase orders and invoices as well as for payments and shipment information.

Here is a simple EDI use case in which Smart Electronics (a fictitious Manufacturing company) uses EDI to send Purchase Orders to Supplier S, and Supplier S sends corresponding invoices to Smart Electronics.

EDI follows specific message standards and formats like EDIFACT or X12 which are the main EDI message standards.

Other EDI standards have been developed for specific purposes or industries, such as:

• ODETTE: used in automotive industry in European countries
• HL7: used for healthcare data.
• IATA PATIS: used to exchange passenger and airport data, and is maintained by the IATA (the International Air Transport Association)
• SWIFT: used for transaction and message exchanges in the financial and banking sectors.