Knowledge Hub

Data Streaming is a method of processing and transmitting data continuously and in real-time rather than in batches. It involves the continuous flow of data from sources to destinations, allowing for the rapid analysis, processing, and dissemination of information.

To understand Data Streaming, suppose data streaming is like water moving in a river on regular basis as compared to a place where water is stored one time with no more coming in. In traditional methods, we collect a bunch of data, like filling a bucket, and then analyze it all at once. On the other hand, in data streaming, it’s like regular pieces of data keep coming in piece by piece, in smaller size, rather than we pull in a huge size in one-go.

Few examples for better understanding.

• Social Media Feeds: Social media platforms use data streaming to show new posts, comments, and updates in real-time. For example, when you refresh your Instagram feed, you instantly see new photos and comments as they are posted.
• Weather Monitoring: Meteorologists use data streaming to receive and analyze continuous weather data. For example, temperature, humidity, and wind speed information are continuously sent and analyzed to provide real-time weather updates.
• Financial Transactions: Banks use data streaming to process and monitor financial transactions in real-time. For example, when you make a purchase with your credit card, the transaction is immediately reflected in your account balance.
• Health Monitoring Devices: Wearable devices use data streaming to provide real-time health information. For example, a fitness tracker continuously measures your heart rate, steps, and sleep patterns, updating your health stats in real-time.
• Traffic Monitoring: Traffic control systems use data streaming to monitor and manage traffic flow. For example, cameras and sensors on roads continuously provide real-time information about traffic conditions, helping optimize traffic signals.
• Online Video Streaming: Services like YouTube or Netflix use data streaming to deliver videos in real-time. For example, when you watch a video online, it starts playing almost instantly, with the data being streamed to your device as you watch.

Batch Processing is a method of handling data in large quantities by grouping it into sets, or batches, and processing those sets together at scheduled intervals. Unlike real-time processing, where data is handled immediately, batch processing involves accumulating data over a period and then processing it as a group.

Think of batch processing like doing your homework assignments at the end of the day rather than right after each class. You collect all the tasks and work on them together. Similarly, in batch processing, data is gathered over time, and then, as a batch, it's processed all at once.

Few examples for better understanding.

• Grade Processing in Schools: Teachers collect students' assignments, quizzes, and tests over a week and then enter grades into the system at the end of the grading period. For example, instead of entering each student's grade immediately after they submit an assignment, the teacher waits and processes grades for the entire class at once.
• Bank Transactions: Banks often update account balances and transactions overnight, processing all the transactions that occurred during the day in a batch. For example, if you deposit money or make a purchase, the transaction might not immediately reflect in your account balance; it's updated in a nightly batch.
• Payroll Processing: Companies often process payroll in batches, calculating and distributing salaries at specific intervals. For example, employee work hours, overtime, and other payroll-related information are collected over a pay period and processed together to generate paychecks.
• Inventory Management: Retailers may update their inventory levels in batches rather than immediately after each sale. For example, Sales data is collected throughout the day, and inventory levels are adjusted during a nightly batch process.
• Billing and Invoicing: Utility companies may process billing information in batches, generating bills for all customers at specific intervals. For example, Instead of generating a bill for each customer as soon as they use a service, bills are created in a batch at the end of the billing cycle.
• Batch Photo Editing: When editing photos, a batch processing approach may be used to apply the same set of edits to multiple photos simultaneously. For example, if you want to resize, crop, or apply a filter to a set of pictures, you can process them all together in a batch.
• Report Generation: Organizations often generate reports based on collected data at specific intervals, such as daily or monthly. For example, Sales reports, customer analytics, or website traffic reports may be generated in batches to provide a comprehensive overview.

Indeed, it's an emerging database model. While there aren't many implementations yet, the concept is promising. If it materializes, it has the potential to streamline operations by eliminating the need for separate SQL and NoSQL datastores, saving significant effort.

Let’s decode it…..

NewSQL is a term that refers to a class of modern relational database management systems (RDBMS) designed to meet the performance and scalability requirements of modern applications, particularly in distributed and cloud computing environments. These databases aim to provide the benefits of traditional relational databases, such as ACID (Atomicity, Consistency, Isolation, Durability) compliance and structured query language (SQL) support, while addressing the challenges posed by large-scale and highly concurrent systems.

Key characteristics and features of NewSQL databases include:

• Scalability: NewSQL databases are designed to scale horizontally, allowing them to handle increasing workloads by adding more nodes to a distributed system.
• Performance: These databases often focus on providing high performance for read and write operations, especially in scenarios where traditional relational databases might face challenges in handling large volumes of data or high transaction rates.
• ACID Compliance: NewSQL databases maintain the ACID properties to ensure data integrity and consistency, crucial for applications where accuracy and reliability are paramount.
• SQL Compatibility: NewSQL databases support traditional SQL queries, making it easier for developers to transition from traditional relational databases to these newer systems.
• Architecture: They may use innovative architectural designs, such as distributed and shared-nothing architectures, to distribute data and workload across multiple nodes.
• Examples of NewSQL databases include:
• Google Spanner: A globally distributed; horizontally scalable database system developed by Google.
• CockroachDB: An open-source, distributed SQL database that is designed to be scalable, resilient, and strongly consistent.
• NuoDB: A distributed SQL database that provides on-demand scalability and continuous availability.

First, I wanted to name this topic as Data or Delta Sharing but to make it more interesting, kept it as Delta Sharing ONLY 😊.

Let’s decode it…..

We all have heard about Data Sharing. It is about sharing data across departments, across branches, across organizations or in DaaS model (explained separately) with the open wide world. In preceding era, Data was not huge, Data was not treated as THE SUCCESS factor, but it felt it is. So, sharing data to other entities was OK, giving them data itself was OK.

But in the current era where data has exponentially grown, and data has become THE SUCCESS factor for any organization. Plus, due to the data size, it’s no longer possible to transfer data even if it is, it’s not in timely manner, secondly due to Data Sovereignty, organizations don’t want their data to move out of their datastores.

Above scenario was even possible in previous era where, to others, we used to give access in our environment but on a very limited basis as accessing our environment or our databases used to impact our performance and used to impact our applications which was not acceptable, that is the reason we started masking or anonymizing data and started exporting actual data to other datastores.

Implementing a robust cybersecurity strategy involves encryption, regular audits, access controls, employee training, and staying updated on emerging threats to safeguard sensitive information effectively.

Few pointers:

• Data Encryption: Employ end-to-end encryption to protect data during transmission and at rest. Utilize strong encryption algorithms to secure sensitive information.
• Access Controls: Implement strict access controls, granting users only the permissions necessary for their roles. Regularly review and update access privileges to ensure they align with current job responsibilities.
• Regular Audits: Conduct periodic security audits to identify vulnerabilities and weaknesses. Regularly assess and update security policies based on audit findings to address new threats.
• Employee Training: Provide comprehensive cybersecurity training for employees to raise awareness about potential threats. Promote a culture of security consciousness to minimize human-related risks, such as phishing attacks.
• Incident Response Plan: Develop and regularly test an incident response plan to respond to security incidents swiftly and effectively. Clearly define roles and responsibilities to minimize downtime during an incident.
• Data Backups: Establish regular and secure data backup procedures to ensure quick recovery in case of data loss or a security breach. Store backups in a remote location to save you from being compromised.
• Network Security: Employ firewalls, intrusion detection and prevention systems to monitor and protect the network. Regularly update and patch software to address known vulnerabilities.
• Vendor Security Assessment: Assess and monitor the cybersecurity measures of third-party vendors handling your data. Ensure vendors comply with security standards and protocols.
• Security Patching: Regularly apply security patches and updates to all systems and software to address known vulnerabilities. Monitor for vulnerabilities that may require immediate attention.
• Data Lifecycle Management: Implement data retention and destruction policies to minimize the risk of unauthorized access. Clearly define how data is collected, processed, stored, and disposed of throughout its lifecycle.

Slowly Changing Dimensions is also known as SCD and is used in Data Warehouse concept. Folks in Data World are very much aware of this term.

Before we understand SCD, one must understand Fact and Dimensions. In dimensional Data Warehouse or in a Data Mart (explained in separate topic), there are two types of schema(s) i.e., Star and Snowflake Schema(s). Both are build based on Facts and Dimensions.

• Facts: This table keeps the transaction’s lowest level of data which can be numeric or non-numeric like sales numbers, number of clicks on social media portal, number of searches in google etc. It’s mainly loaded in append mode, means rows always get inserted. This table is a very fast-moving table, and rows keep coming in. There are hardly any organizations which update and delete data from Fact Tables.
• Dimension: This table on the other hand keeps data element by which Fact can be analyzed e.g., sale by time, sales by region, sales by the product etc. So, Time, Regions, Products etc., are dimensions. In contrast with Fact table, frequency of rows coming in these tables are very low. For Example, once we have loaded regions then when often regions will change 😊. Once Time is loaded, it will never change. At the same time, Products definitions can change. A company today has 100 products, in future it can have more. Then attributes of one product can change in future a well, so there are three ways to keep the history of changed in Dimensions which are handled by principles managed by SLOWLY CHANGING DIMENSIONS (SCD).

The concept of text and embedded vectorization in machine learning can be a bit confusing when compared to general data or text vectorization. Both approaches share the overarching goal of accelerating tasks, but they operate in different ways.

General Vectorization: The primary aim is to convert a single action on a single data element (SISD) into a single action on multiple data elements. This involves transforming the data in a way that allows for parallel processing, improving efficiency in handling multiple data elements simultaneously.

Text Vectorization in Machine Learning: In the context of machine learning, text vectorization specifically deals with converting textual or character data into numerical representations. Once the text or characters are converted into numbers, the embedding process comes into play. This process involves using machine learning and deep learning techniques to learn from these numerical representations, facilitating faster and more effective results.

Many folks are getting confused between Concurrent Processing or MPP (Massive Parallel Processing).

Please note there is a major difference between both where one is dependent on the developers and the other on hardware.

Concurrent Processing means when technology can run concurrent queries which most of the time dependent on cores handling the queries.

In simple words, when more than one job runs in parallel and execute some code sets at the same time.

On the other hand, MPP utilizes hardware level concurrency where one job/ code set is divided in multiple pieces and runs in parallel mode.

In simple words, a single job utilizes multiple nodes to execute the same code set making it much faster as compared to job utilize single node at a time.

Having Concurrent and MPP capability gives a huge benefit to the solution when it comes to meeting SLA(s). In Big Data era, having these capabilities is a MUST-TO-HAVE.

Information Modelling is not a new term, but nerd data folks hardly get in touch with it as it’s one level above of creating any special solution.

Let’s decode it…..

We also know when we start a project or solution, we begin with Conceptual Model then Logical Model then Physical Model etc. Unlike these models where we focus especially on either entities or relationships or process or constraints, Information Modelling represents entities, relationships, and processes within a system or organization.

Information modelling is a crucial aspect of designing data solutions that goes beyond merely structuring databases. It involves creating a comprehensive representation of how information is organized, flows, and is utilized within an organization or system. Information models provide a framework for understanding the relationships and dependencies among different elements of information, facilitating effective communication and management.

These are one of the most confusing topics. We all mix-up data quality fixes and standardization of data for better processing with transforming data for better processing and serving.

Let’s decode it…..

We all know, in Data Warehouse or when we pull data from an OLTP system, we first store it in a LANDING ZONE somewhere which normally is the true copy of source systems. Then we move it to STAGING ZONE before we transform it into a Unified Data Warehouse (Bill Inmon way) or in a Dimensional Model (Ralph Kimble way).

Tables in LANDING and STANGING ZONES, both keep the true DATA MODEL from source systems but not the CONDITION of DATA.

WHAT DOES ABOVE STATEMENT MEANS, not the CONDITION of DATA?

Data Standardization

Does standardizing the condition of data not equal to transformation? NOPS, ITS NOT. Data Standardization can have unlimited scenarios, few are shared below e.g.,

• Standardization of Dates:
  • Converting different dates formats from different source systems e.g., DD/MM/YY, MM/DD/YY, YY/MM/DD, YY/DD/MM etc., into one format e.g., DD/MM/YYYY.
  • Convert empty date rows into e.g., 01/01/9999.
• Convert special characters to NA.
• Convert empty fields to NA.
• Profile the fields, if data type is string or varchar but data is only number than convert it into digit or number as running queries on digit or numbers are way faster.
• If the field contains codes only with only character, then make it CHAR and if it only contains numbers then make it DIGIT.
• If the full name of a person is in one field, convert it in First, Middle and Last Name.
• Fix the spelling mistakes.
• Fix the up and lower cases e.g., if a country is Australia and in data its coming as Aus, Aust, australia, Aussie, aus etc., than convert it in one format e.g., Australia.