Knowledge Hub

Information Modelling is not a new term, but nerd data folks hardly get in touch with it as it’s one level above of creating any special solution.

Let’s decode it…..

We also know when we start a project or solution, we begin with Conceptual Model then Logical Model then Physical Model etc. Unlike these models where we focus especially on either entities or relationships or process or constraints, Information Modelling represents entities, relationships, and processes within a system or organization.

Information modelling is a crucial aspect of designing data solutions that goes beyond merely structuring databases. It involves creating a comprehensive representation of how information is organized, flows, and is utilized within an organization or system. Information models provide a framework for understanding the relationships and dependencies among different elements of information, facilitating effective communication and management.

These are one of the most confusing topics. We all mix-up data quality fixes and standardization of data for better processing with transforming data for better processing and serving.

Let’s decode it…..

We all know, in Data Warehouse or when we pull data from an OLTP system, we first store it in a LANDING ZONE somewhere which normally is the true copy of source systems. Then we move it to STAGING ZONE before we transform it into a Unified Data Warehouse (Bill Inmon way) or in a Dimensional Model (Ralph Kimble way).

Tables in LANDING and STANGING ZONES, both keep the true DATA MODEL from source systems but not the CONDITION of DATA.

WHAT DOES ABOVE STATEMENT MEANS, not the CONDITION of DATA?

Data Standardization

Does standardizing the condition of data not equal to transformation? NOPS, ITS NOT. Data Standardization can have unlimited scenarios, few are shared below e.g.,

• Standardization of Dates:
  • Converting different dates formats from different source systems e.g., DD/MM/YY, MM/DD/YY, YY/MM/DD, YY/DD/MM etc., into one format e.g., DD/MM/YYYY.
  • Convert empty date rows into e.g., 01/01/9999.
• Convert special characters to NA.
• Convert empty fields to NA.
• Profile the fields, if data type is string or varchar but data is only number than convert it into digit or number as running queries on digit or numbers are way faster.
• If the field contains codes only with only character, then make it CHAR and if it only contains numbers then make it DIGIT.
• If the full name of a person is in one field, convert it in First, Middle and Last Name.
• Fix the spelling mistakes.
• Fix the up and lower cases e.g., if a country is Australia and in data its coming as Aus, Aust, australia, Aussie, aus etc., than convert it in one format e.g., Australia.

Semantics are different for everyone. In an organization, each department has its own semantics, to view data. Same data can have different meanings for one department as compared to others.

Salary for HR department is to know who is getting what and when they hire new candidate, they can refer to what others are getting based on role, JD, skill set etc.. But Salary for Finance department can be, to calculate annual expense for the company.

Semantic Layer in Decision Support Systems are based on business requirements, in their language, using their own key words, their codes etc.

Interoperability on the other hand is about exchanging something across systems so others don’t need to regenerate it again and again.

Semantic Interoperability is about exchanging semantic information across systems or across departments or across organizations. But this comes with a challenge whether the receiving party can understand the keywords, symbols, terminologies etc., from the sending party.

Approaches play a vital role in the failure of many projects in the IT World. We all have heard about POC, Protype, Pilot, MVP, SIT, UAT, and Production implementations, but I am sure, most of us might have not given a thorough thought of aftereffects, if one is interchanged with other.

Let’s try to understand everything in the upcoming topics but to give a broader perspective all above approaches have their own implementation strategies. Most of the projects fail when e.g., started with a POC and once the POC is successful, move the same in Production. One must not do that. This normally happens when business push implementers to achieve tight timelines.

Please note, small level of implementation which in our example, POC might be successful with limited infrastructure or software, but 99% chances are when it goes in Production, it’s bound to fail. Same is true for Prototype, Pilot and MVP environments. One must plan infrastructure and software based on, if plans to be in Production.

POC stands for Proof of Concept. POC normally follows Sales Context of the product.

In POC, output is most important, not technology. In POC, the implementers make sure output is fulfilling customer requirements and at the same time convince business that this is the solution to go with.

The Goal of POC is to make sure whether Idea is Feasible or not.

‘POC is built and run for Business Users.’

Protype is the next stage of POC. In POC, one has test that yes Idea is Feasible, and it will benefit business agenda.

Now, in Prototyping, tools and technologies comes into picture where one starts converting Ideas in Reality, where it will show how the end-result will look like. It will show, how product will look like. It will show the real end solution.

Prototype is not built considering that business users will jump in and start using it. Prototypes are normally for technical folks to evaluate the end results with dumpy but relevant data.

’Prototype is built and run for Technical Users.’

New name of Pilot is MVP 😊. MVP is the next stage of Prototyping where Tools and Technologies are tested, and end solution is tested by technical folks. Now in MVP, subset of actual requirements is considered, subset of real data is used, subset of hardware and infrastructure is used, subset of software is used, and limited number of features are implemented.

Once MVP is ready, it’s released to actual users to start using it. This gives business users a taste of the actual solution. The biggest advantage of this approach is, with small implementation, with small investment, business can taste the blood and take a decision to go to the next stage or drop the solution.

Now a days, many organizations divide complete solution into multiple MVP(s) i.e., by implementing solutions in chunks, this is also supported by Agile approach where product gets ready in shorter timeframes and then keeps maturing with time. In MVP, just like actual implementation, three environments are spin i.e., DEV, UAT and PROD, along with all the steps like DevOps, CI/CD, Metadata Management, Data Quality, Data Governance, Data Lineage, Master and Reference Data Management, and many other steps which are required to go to Production.

In MVP stage where actual Tools and Technologies, and SIT, UAT, and PRODUCTION environments are implemented on a small scale.

Once MVP is successful, the next stage is to gather complete business requirements, a complete designed is made, teams are built, project plan is made, end-to-end SOW is signed, development is conducted, testing is done i.e., UT, SIT and UAT, and solution goes live.

On contrast with POC, Prototype and Pilot/ MVP, in this stage a complete Infrastructure, Software and Data is prepared and used.

UT stands for Unit testing and is done by developer during development phase for every piece of code developed. The better the UT is, the better the product will be.

SIT stands for Systems Integration Testing. Now a days, SIT is disappearing, SIT is the first step to save cost which is NOT a good idea. I am sure in a decade, newbies in IT World might not even know what actually SIT is. Once complete solution is ready, it’s moved to another environment which do have all the possible components of the actual solution but in term of size, it is small e.g., if based on data there is 10 servers and 100 licenses are required, SIT can be run with 1 server and with 1 license. In SIT, we make sure code can run end-to-end, user access is properly set, security protocols are in-place, jobs dependencies are in-place etc.

CRUD and CRAP functions are our daily used processes for data folks, but surely by name most are not aware of these words 😊.

Let’s decode these…..

As we have been using structured data for the last few decades, we know how to create, read, update, and delete. These basic functions which we use in any database are called CRUD data.

Now, the new era, the current era, surfaced where data started to flow-in with high velocity from web logs, social streams, CCTV, sensor data, videos, Facebook, or Tweeter data, mobile, geo-spatial, and so on. All this data has nothing to do with transactional data, so we used to call it CRAP data 😊. But now these have become one of the most valuable assets of any organization. The speed this data is coming, it’s normally not possible to update or delete it so it’s been created, replicate, appended, and process only.

CRUD stands for Create, Read, Update, and Delete

Accessibility and Visibility are often used interchangeably, especially in simple terms. However, in the world of data solutions, it's crucial to distinguish between them.

Let’s decide it…..

Accessibility pertains to your ability to connect to and utilize data for decision-making purposes.

On the other hand, Visibility involves having a comprehensive understanding of what's happening behind the scenes with your data.

For instance, if you request your Data Warehousing or Data Lakehouse datastore owners to extract data from source systems and transform it into the desired format for decision-making, two key concepts come into play:

Accessibility: This is achieved when your database administrator or datastore owner grants you privileges within the database, allowing you to read, write, or update your data.

Visibility: This extends beyond mere access. It encompasses a deep understanding of various aspects of your data, such as what was sourced, the business rules used for transformations, the presence of business, technical, or operational metadata, the formula for golden records, data quality assurance methods, data loading approaches (e.g., CDC, batch, real-time), SLAs, and lineage information, among other things. All of these elements collectively provide you with comprehensive Visibility into your data.