Knowledge Hub

Another often overlooked subject is particularly significant in the age of Big Data, where the substantial influence of unstructured data on cloud storage costs is evident. In this era, recognizing the importance of data compression is essential; neglecting its utilization translates to unnecessary expenditures and extended return on investment timelines.

Let’s decode it…..

Data compression is a method of decreasing the size of files or data to save storage space and communication bandwidth. It involves encoding information using fewer bits than the original representation.

Here's a simple explanation with examples:

• Text Compression:
  • Imagine you have a text file with the sentence: "The quick brown fox jumps over the lazy dog."
  • Without compression, each character is represented by a certain number of bits (usually 8 bits per character in ASCII). A lot of space can be taken up by this.
  • Through compression, you can replace commonly used words or phrases with shorter codes. For instance, you might replace "the quick brown fox jumps over" with a single code. When decompressed, the original sentence is reconstructed.
• Image Compression:
  • Consider a digital image with millions of pixels, each represented by color information (e.g., RGB values).
  • Compression techniques reduce redundancy in the image. Instead of storing each pixel's color separately, you might store patterns or use mathematical transformations to represent the image more efficiently.
  • JPEG is a common image compression format that achieves significant file size reduction while maintaining reasonable image quality.
• Audio Compression:
  • In the case of audio, there is a vast amount of data in a music file, especially in high-quality formats.
  • Audio compression algorithms, like MP3, remove redundant or less perceptually important information. They take advantage of the fact that our ears are less sensitive to certain frequencies and remove data that may not be noticeable to the listener.
• Lossless vs. Lossy Compression:
  • Lossless compression retains all the original data when decompressed. It's like zipping a file and then unzipping it, what you get back is exactly the same as what you started with.
  • Lossy compression sacrifices some information to achieve higher compression ratios. For example, in JPEG compression, some details in an image might be lost, but the trade-off is a much smaller file size.
• Compression in Everyday Technology:
  • When you download a ZIP file from the internet, you're using a form of data compression. The ZIP file contains your files in a compressed form, and when you unzip it, the original files are restored.
  • Streaming services use compression to transmit audio and video efficiently, allowing you to watch videos or listen to music without having to download huge files.

Data Warehouse

A data warehouse is a type of data platform that is designed to store and analyze structured data from various sources, providing a central repository for business intelligence (BI) and analytics applications. The data stored in a data warehouse is typically historical, integrated, and cleansed, and can be used to support decision-making, reporting, and analysis at all levels of an organization. The data stored in a data warehouse is typically structured and organized into tables and is optimized for efficient querying and analysis.

Data Lake

A data lake is a centralized repository that allows you to store all your structured, semi and unstructured data at any scale. The term was coined to represent the idea of a system where data flows in from various sources and resides in its most "natural" state, just like water in a lake. These data can range from raw, unprocessed files to transformed, structured records.

In general terms, deactivation often implies the action of making something inactive or turning it off. In the context of data, this refers to the intentional process of rendering certain data inactive, which involves restricting access, archiving, or taking it out of active use. Here are a few possible interpretations, but it's important to consider the specific context in which you encountered the term:

• Archiving or Disabling Data: Data deactivation refers to archiving data or making it temporarily inactive. This could be done for various reasons, such as compliance with data retention policies or reducing the storage load.
• User Account Deactivation: In the context of user accounts, data deactivation means disabling or deactivating a user account, preventing further access or updates to associated data.
• Temporary Suspension: It refers to a temporary suspension or deactivation of certain data processing activities for maintenance, security, or other reasons.
• Here are a few examples of scenarios where a similar concept might be applied:
• Archiving Inactive Customer Records: In a customer database, records of customers who haven't engaged with the business for an extended period might be archived or deactivated. This involves making the data inactive, but it is still retained for historical or compliance purposes.
• Account Deactivation: When an employee leaves a company, their user account in the company's systems may be deactivated. This deactivation prevents the former employee from accessing company data and systems.
• Temporary Suspension for Maintenance: A web application might temporarily deactivate certain features or services for maintenance or updates. During this period, certain data-related functionalities could be temporarily disabled.
• Data Retention Policies: Organizations might have data retention policies that specify when certain types of data should be deactivated, archived, or permanently deleted. This ensures compliance with regulations and reduces storage costs.
• Disabling Inactive Products or Services: In an e-commerce system, products or services that are no longer offered or are inactive may be flagged as such. This deactivation prevents customers from making new purchases of those items.
• Temporary Data Deactivation for Security Reasons: In the event of a security incident, certain data access might be temporarily deactivated to prevent unauthorized access. This could involve suspending certain user accounts or restricting access to sensitive information.

In simple terms, when we talk about an Embargo, we're usually talking about a rule or limit, especially when it comes to trading or sharing information. Now, if we use "data embargo," it means there are rules or limits on using, sharing, or spreading certain information. This mostly happens for legal reasons, to keep things safe, to protect privacy, or because there are rules to follow.

Let's delve into the concept of an "embargo," especially in the context of data, using simpler language:

1. Embargo Basics: An "embargo" is like a rule that says you can't do certain things. It often happens in trade or when sharing information.
2. Data Embargo: Now, if we add "data" to the word "embargo," it means there's a rule or limit on how we can use, share, or spread specific information. Imagine it's like having a set of rules for how you can talk about or share certain details.

These rules could be there for different reasons:

Data Streaming is a method of processing and transmitting data continuously and in real-time rather than in batches. It involves the continuous flow of data from sources to destinations, allowing for the rapid analysis, processing, and dissemination of information.

To understand Data Streaming, suppose data streaming is like water moving in a river on regular basis as compared to a place where water is stored one time with no more coming in. In traditional methods, we collect a bunch of data, like filling a bucket, and then analyze it all at once. On the other hand, in data streaming, it’s like regular pieces of data keep coming in piece by piece, in smaller size, rather than we pull in a huge size in one-go.

Few examples for better understanding.

• Social Media Feeds: Social media platforms use data streaming to show new posts, comments, and updates in real-time. For example, when you refresh your Instagram feed, you instantly see new photos and comments as they are posted.
• Weather Monitoring: Meteorologists use data streaming to receive and analyze continuous weather data. For example, temperature, humidity, and wind speed information are continuously sent and analyzed to provide real-time weather updates.
• Financial Transactions: Banks use data streaming to process and monitor financial transactions in real-time. For example, when you make a purchase with your credit card, the transaction is immediately reflected in your account balance.
• Health Monitoring Devices: Wearable devices use data streaming to provide real-time health information. For example, a fitness tracker continuously measures your heart rate, steps, and sleep patterns, updating your health stats in real-time.
• Traffic Monitoring: Traffic control systems use data streaming to monitor and manage traffic flow. For example, cameras and sensors on roads continuously provide real-time information about traffic conditions, helping optimize traffic signals.
• Online Video Streaming: Services like YouTube or Netflix use data streaming to deliver videos in real-time. For example, when you watch a video online, it starts playing almost instantly, with the data being streamed to your device as you watch.

Indeed, it's an emerging database model. While there aren't many implementations yet, the concept is promising. If it materializes, it has the potential to streamline operations by eliminating the need for separate SQL and NoSQL datastores, saving significant effort.

Let’s decode it…..

NewSQL is a term that refers to a class of modern relational database management systems (RDBMS) designed to meet the performance and scalability requirements of modern applications, particularly in distributed and cloud computing environments. These databases aim to provide the benefits of traditional relational databases, such as ACID (Atomicity, Consistency, Isolation, Durability) compliance and structured query language (SQL) support, while addressing the challenges posed by large-scale and highly concurrent systems.

Key characteristics and features of NewSQL databases include:

• Scalability: NewSQL databases are designed to scale horizontally, allowing them to handle increasing workloads by adding more nodes to a distributed system.
• Performance: These databases often focus on providing high performance for read and write operations, especially in scenarios where traditional relational databases might face challenges in handling large volumes of data or high transaction rates.
• ACID Compliance: NewSQL databases maintain the ACID properties to ensure data integrity and consistency, crucial for applications where accuracy and reliability are paramount.
• SQL Compatibility: NewSQL databases support traditional SQL queries, making it easier for developers to transition from traditional relational databases to these newer systems.
• Architecture: They may use innovative architectural designs, such as distributed and shared-nothing architectures, to distribute data and workload across multiple nodes.
• Examples of NewSQL databases include:
• Google Spanner: A globally distributed; horizontally scalable database system developed by Google.
• CockroachDB: An open-source, distributed SQL database that is designed to be scalable, resilient, and strongly consistent.
• NuoDB: A distributed SQL database that provides on-demand scalability and continuous availability.

First, I wanted to name this topic as Data or Delta Sharing but to make it more interesting, kept it as Delta Sharing ONLY 😊.

Let’s decode it…..

We all have heard about Data Sharing. It is about sharing data across departments, across branches, across organizations or in DaaS model (explained separately) with the open wide world. In preceding era, Data was not huge, Data was not treated as THE SUCCESS factor, but it felt it is. So, sharing data to other entities was OK, giving them data itself was OK.

But in the current era where data has exponentially grown, and data has become THE SUCCESS factor for any organization. Plus, due to the data size, it’s no longer possible to transfer data even if it is, it’s not in timely manner, secondly due to Data Sovereignty, organizations don’t want their data to move out of their datastores.

Above scenario was even possible in previous era where, to others, we used to give access in our environment but on a very limited basis as accessing our environment or our databases used to impact our performance and used to impact our applications which was not acceptable, that is the reason we started masking or anonymizing data and started exporting actual data to other datastores.

CDC framework is much more than CDC as a feature or as a capability by a tool. There are many tools in the market which can just extract CDC data from database which is only the first step in whole CDC framework.

CDC: Full abbreviation of CDC is Change Data Capture, meaning the moment, the data is changed in database ONLY THE CHANGED DATA gets picked up by the CDC process and pushed to next layer. Changed data can be like 1) a new row can be added or 2) an existing row can be updated or 3) a previously added row can be deleted or dropped.

This whole process can be Real-Time but mostly can't as Real-Time + CDC needs a lot of memory processing which comes with heavy TCO, so it all depends on ROI for the related Business Case.

CDC Framework: Extracting only changed data is not the end of settling data to be presented in Dashboards or Reporting, there are two more steps one must complete before changed data is considered useful.

Slowly Changing Dimensions is also known as SCD and is used in Data Warehouse concept. Folks in Data World are very much aware of this term.

Before we understand SCD, one must understand Fact and Dimensions. In dimensional Data Warehouse or in a Data Mart (explained in separate topic), there are two types of schema(s) i.e., Star and Snowflake Schema(s). Both are build based on Facts and Dimensions.

• Facts: This table keeps the transaction’s lowest level of data which can be numeric or non-numeric like sales numbers, number of clicks on social media portal, number of searches in google etc. It’s mainly loaded in append mode, means rows always get inserted. This table is a very fast-moving table, and rows keep coming in. There are hardly any organizations which update and delete data from Fact Tables.
• Dimension: This table on the other hand keeps data element by which Fact can be analyzed e.g., sale by time, sales by region, sales by the product etc. So, Time, Regions, Products etc., are dimensions. In contrast with Fact table, frequency of rows coming in these tables are very low. For Example, once we have loaded regions then when often regions will change 😊. Once Time is loaded, it will never change. At the same time, Products definitions can change. A company today has 100 products, in future it can have more. Then attributes of one product can change in future a well, so there are three ways to keep the history of changed in Dimensions which are handled by principles managed by SLOWLY CHANGING DIMENSIONS (SCD).

The concept of text and embedded vectorization in machine learning can be a bit confusing when compared to general data or text vectorization. Both approaches share the overarching goal of accelerating tasks, but they operate in different ways.

General Vectorization: The primary aim is to convert a single action on a single data element (SISD) into a single action on multiple data elements. This involves transforming the data in a way that allows for parallel processing, improving efficiency in handling multiple data elements simultaneously.

Text Vectorization in Machine Learning: In the context of machine learning, text vectorization specifically deals with converting textual or character data into numerical representations. Once the text or characters are converted into numbers, the embedding process comes into play. This process involves using machine learning and deep learning techniques to learn from these numerical representations, facilitating faster and more effective results.