Knowledge Hub

CDC framework is much more than CDC as a feature or as a capability by a tool. There are many tools in the market which can just extract CDC data from database which is only the first step in whole CDC framework.

CDC: Full abbreviation of CDC is Change Data Capture, meaning the moment, the data is changed in database ONLY THE CHANGED DATA gets picked up by the CDC process and pushed to next layer. Changed data can be like 1) a new row can be added or 2) an existing row can be updated or 3) a previously added row can be deleted or dropped.

This whole process can be Real-Time but mostly can't as Real-Time + CDC needs a lot of memory processing which comes with heavy TCO, so it all depends on ROI for the related Business Case.

CDC Framework: Extracting only changed data is not the end of settling data to be presented in Dashboards or Reporting, there are two more steps one must complete before changed data is considered useful.

Slowly Changing Dimensions is also known as SCD and is used in Data Warehouse concept. Folks in Data World are very much aware of this term.

Before we understand SCD, one must understand Fact and Dimensions. In dimensional Data Warehouse or in a Data Mart (explained in separate topic), there are two types of schema(s) i.e., Star and Snowflake Schema(s). Both are build based on Facts and Dimensions.

• Facts: This table keeps the transaction’s lowest level of data which can be numeric or non-numeric like sales numbers, number of clicks on social media portal, number of searches in google etc. It’s mainly loaded in append mode, means rows always get inserted. This table is a very fast-moving table, and rows keep coming in. There are hardly any organizations which update and delete data from Fact Tables.
• Dimension: This table on the other hand keeps data element by which Fact can be analyzed e.g., sale by time, sales by region, sales by the product etc. So, Time, Regions, Products etc., are dimensions. In contrast with Fact table, frequency of rows coming in these tables are very low. For Example, once we have loaded regions then when often regions will change 😊. Once Time is loaded, it will never change. At the same time, Products definitions can change. A company today has 100 products, in future it can have more. Then attributes of one product can change in future a well, so there are three ways to keep the history of changed in Dimensions which are handled by principles managed by SLOWLY CHANGING DIMENSIONS (SCD).

The concept of text and embedded vectorization in machine learning can be a bit confusing when compared to general data or text vectorization. Both approaches share the overarching goal of accelerating tasks, but they operate in different ways.

General Vectorization: The primary aim is to convert a single action on a single data element (SISD) into a single action on multiple data elements. This involves transforming the data in a way that allows for parallel processing, improving efficiency in handling multiple data elements simultaneously.

Text Vectorization in Machine Learning: In the context of machine learning, text vectorization specifically deals with converting textual or character data into numerical representations. Once the text or characters are converted into numbers, the embedding process comes into play. This process involves using machine learning and deep learning techniques to learn from these numerical representations, facilitating faster and more effective results.

Many folks are getting confused between Concurrent Processing or MPP (Massive Parallel Processing).

Please note there is a major difference between both where one is dependent on the developers and the other on hardware.

Concurrent Processing means when technology can run concurrent queries which most of the time dependent on cores handling the queries.

In simple words, when more than one job runs in parallel and execute some code sets at the same time.

On the other hand, MPP utilizes hardware level concurrency where one job/ code set is divided in multiple pieces and runs in parallel mode.

In simple words, a single job utilizes multiple nodes to execute the same code set making it much faster as compared to job utilize single node at a time.

Having Concurrent and MPP capability gives a huge benefit to the solution when it comes to meeting SLA(s). In Big Data era, having these capabilities is a MUST-TO-HAVE.

Information Modelling is not a new term, but nerd data folks hardly get in touch with it as it’s one level above of creating any special solution.

Let’s decode it…..

We also know when we start a project or solution, we begin with Conceptual Model then Logical Model then Physical Model etc. Unlike these models where we focus especially on either entities or relationships or process or constraints, Information Modelling represents entities, relationships, and processes within a system or organization.

Information modelling is a crucial aspect of designing data solutions that goes beyond merely structuring databases. It involves creating a comprehensive representation of how information is organized, flows, and is utilized within an organization or system. Information models provide a framework for understanding the relationships and dependencies among different elements of information, facilitating effective communication and management.

These are one of the most confusing topics. We all mix-up data quality fixes and standardization of data for better processing with transforming data for better processing and serving.

Let’s decode it…..

We all know, in Data Warehouse or when we pull data from an OLTP system, we first store it in a LANDING ZONE somewhere which normally is the true copy of source systems. Then we move it to STAGING ZONE before we transform it into a Unified Data Warehouse (Bill Inmon way) or in a Dimensional Model (Ralph Kimble way).

Tables in LANDING and STANGING ZONES, both keep the true DATA MODEL from source systems but not the CONDITION of DATA.

WHAT DOES ABOVE STATEMENT MEANS, not the CONDITION of DATA?

Data Standardization

Does standardizing the condition of data not equal to transformation? NOPS, ITS NOT. Data Standardization can have unlimited scenarios, few are shared below e.g.,

• Standardization of Dates:
  • Converting different dates formats from different source systems e.g., DD/MM/YY, MM/DD/YY, YY/MM/DD, YY/DD/MM etc., into one format e.g., DD/MM/YYYY.
  • Convert empty date rows into e.g., 01/01/9999.
• Convert special characters to NA.
• Convert empty fields to NA.
• Profile the fields, if data type is string or varchar but data is only number than convert it into digit or number as running queries on digit or numbers are way faster.
• If the field contains codes only with only character, then make it CHAR and if it only contains numbers then make it DIGIT.
• If the full name of a person is in one field, convert it in First, Middle and Last Name.
• Fix the spelling mistakes.
• Fix the up and lower cases e.g., if a country is Australia and in data its coming as Aus, Aust, australia, Aussie, aus etc., than convert it in one format e.g., Australia.

Semantics are different for everyone. In an organization, each department has its own semantics, to view data. Same data can have different meanings for one department as compared to others.

Salary for HR department is to know who is getting what and when they hire new candidate, they can refer to what others are getting based on role, JD, skill set etc.. But Salary for Finance department can be, to calculate annual expense for the company.

Semantic Layer in Decision Support Systems are based on business requirements, in their language, using their own key words, their codes etc.

Interoperability on the other hand is about exchanging something across systems so others don’t need to regenerate it again and again.

Semantic Interoperability is about exchanging semantic information across systems or across departments or across organizations. But this comes with a challenge whether the receiving party can understand the keywords, symbols, terminologies etc., from the sending party.

Approaches play a vital role in the failure of many projects in the IT World. We all have heard about POC, Protype, Pilot, MVP, SIT, UAT, and Production implementations, but I am sure, most of us might have not given a thorough thought of aftereffects, if one is interchanged with other.

Let’s try to understand everything in the upcoming topics but to give a broader perspective all above approaches have their own implementation strategies. Most of the projects fail when e.g., started with a POC and once the POC is successful, move the same in Production. One must not do that. This normally happens when business push implementers to achieve tight timelines.

Please note, small level of implementation which in our example, POC might be successful with limited infrastructure or software, but 99% chances are when it goes in Production, it’s bound to fail. Same is true for Prototype, Pilot and MVP environments. One must plan infrastructure and software based on, if plans to be in Production.

POC stands for Proof of Concept. POC normally follows Sales Context of the product.

In POC, output is most important, not technology. In POC, the implementers make sure output is fulfilling customer requirements and at the same time convince business that this is the solution to go with.

The Goal of POC is to make sure whether Idea is Feasible or not.

‘POC is built and run for Business Users.’

Protype is the next stage of POC. In POC, one has test that yes Idea is Feasible, and it will benefit business agenda.

Now, in Prototyping, tools and technologies comes into picture where one starts converting Ideas in Reality, where it will show how the end-result will look like. It will show, how product will look like. It will show the real end solution.

Prototype is not built considering that business users will jump in and start using it. Prototypes are normally for technical folks to evaluate the end results with dumpy but relevant data.

’Prototype is built and run for Technical Users.’