Knowledge Hub

By now, we all know what Semantic Layer is, IT defines data elements for business ease. Then there came the era for Business folks defines data elements themselves based on their own understanding and now the latest approach is going towards Machine Learning Models define data element based on open-ended requirements.

A major difference between first two approaches vs Augmented Analytics is, first two were derived by humans either by IT or Business folks so there is always a factor of human era and at the same time human can be biased base on knowledge and information they possess about the business domain. On the other hand, Augmented Analytics is based on automated Machine Learning which learns the data elements while new data sets keep coming in.

The first approach was static as requirements were always coming from business and implemented by IT folks, so any change had a round robin effect.

Data Island has kind of disappeared as integration technologies have matured.

‘Data Island is a datastore(s) which is/ are not accessible.’

In previous eras when network security was a big concern or even before, when network connectivity was not even there, the datastores were considered as Data Island as external integration was not possible.

Let’s try to understand with examples. We all know WIFI has become a norm in our lives and living without it is like living without Air right? In past when we used to visit a friend’s place we used to greet first, now we first ask, ‘What’s your WIFI Password’ lolz 😊?’. But if you are as old as I am, before 2005 WIFI was not a thing. There were ONLY cables which used to connect our desktops or laptops with network, well I still prefer cable. During that era consider a datastore sitting in different city or region then how to connect? In that situation, we had floppy drives for small data transfer (yeah, I won’t explain what is floppy drive lolz 😊, old folks know what I am talking about. Let’s let newbies find out what is floppy drive), and for huge data we used to have big hard disks and sometimes desktops itself were moved to move data from one location to another location. This datastore, which was not connectable with other datastore, was call Data Island.

Data Silos is always confused with Data Island.

Data Islands are those datastores which are not accessible from external world may be due technology barriers, but Data Silos are those which are only accessible for their relevant stakeholders.

There can be a lot of reasons why data is in silo.

• A particular datastore might be hosting confidential data e.g., finance data, audit data etc.
• A particular data which is not readable e.g., semi or unstructured data sets.
• There is Security or Privacy concerns, so data is not shared among another departments.
• The quality of data is so low that it can’t be shared with management as it may cause wrong decisions and may damage the company’s image.
• Sometime, in development environment different database instances are created for different development teams so if one changes data or table’s structures, other development teams are not impacted.

Why should Data Silos not exist?

There are many ways to achieve data privacy like data generalization, data swapping, data masking, data encryption etc., all these techniques change original data which of course is not desired by most. All these are done to ensure data should not fall into wrong hands.

Data Synthetic is something which will really solve the problem statement like anonymize data before sharing with developers, data science teams for data models etc.

Data Synthetic is about generating new dataset using computer simulations. This dataset contains fake data with the right properties or attributes. For example, customer information is required in almost every application or system, but it can’t be shared to protect individual identity. Data Synthetic tools create a dummy dataset for customers with name, email, contact number, family details etc. but nothing related to original data. This way, developers can build the data pipelines and data scientists can build their data models without thinking of breaching any regulatory or compliance clauses.

There are four types of instructions to pull data i.e., SISD, SIMD, MISD and MIMD.

• SISD: Single Instruction Stream and Single Data Stream
• SIMD: Single Instruction Stream and Multiple Data Stream
• MISD: Multiple Instruction Stream and Single Data Stream
• MIMD: Multiple Instruction Stream and Multiple Data Stream

Before we jump into understanding all above mentioned techniques, let’s first understand what is a Core in a machine or in a computer or in a server etc.?

A CPU Core is built of billions of microscopic transistors to process instructions passed to it. One instruction is handled by One core, if someone needs multitasking or multiple instruction in parallel, minimum 2 cores are required.

We know when we insert or load data in a computer or server or in a node, it gets saved in hard disk. We won’t go in to details how it is stored on the hard disk. We can imagine when we write something on paper, consider the same way, data is saved in a hard disk.

Data Vectorization is something very common nowadays, especially since the inception of Big Data or Hadoop. It’s not like it was not in-use in the past, but I would say it was not famous.

All of us must have heard about MPP (Massive Parallel Processing) right? But do we know how it works at the back end? Data Vectorization is about enabling parallel processing to fetch data.

To understand MPP, let’s refer to four types of instructions to pull data in computer architecture i.e., SISD, SIMD, MISD and MIMD (all explained in separate topic). In this topic, we will be referring to SISD and SIMD only.

Traditionally, a computer or machine or a server works in SISD mode i.e., Single Instruction Stream and Single Data Stream means each instruction fetches required data one by one.

In Data Vectorization, we change our approach to MPP mode i.e., computer or machine or server starts working in SIMD mode i.e., Single Instructure and Multiple Data, means if one query is executed and data is residing in multiple data nodes, data from all nodes will be pulled in parallel making computation must faster as compared to SISD.

Before we understand Data Due Diligence, let’s first have a quick refer what is Due Diligence.

Due diligence is an enquiry, inspect, or assessment executed to check evidence or specifics of a matter beneath attention.

For example, when we are buying something we first check e.g., it is made of which country, it’s a new or used item, it’s produced by which company, what’s the quality of the items etc. All these checks and balances come under due diligence process.

Wikipedia

‘Due diligence is the investigation or exercise of care that a reasonable business or person is normally expected to take before entering into an agreement or contract with another party or an act with a certain standard of care.’

There can be many sub processes involved based on domains, businesses, or solution requirements.

A merger and acquisitions (M&A) process is very old process where due diligence is a must, of course when we’re merging or buying a million-dollar company, we need to make sure our decision is correct.

Data Maturity is the key to becoming a Data Driven Organization. There are many factors to judge it.

‘Maturity refers to stability level of an entity or a process or a product.’

Maturity is not about how many stages of process has passed through, it’s about how stable the product is after implementation.

You must have heard people referring to some individual that he is a mature person so let him take his own decision. Same way, decision can be mature or immature which depend on the result from those decisions.

In this era where Data has become the most powerful input for any organization, well it has been since its start. But now technologies are so MATURE that insight from data analytics are taken very seriously and million-dollar or sometimes billion-dollar deals are taken based on Data Maturity.

Everyone remembers the acquisition of WhatsApp right? Multi-billion-dollar deal just based on how many users is using it and what is the potential growth on a simple mobile application.

As much as Data Filtering is simple as much as it’s so important in Data World.

‘Data Filtering is about reducing the number of rows while pull data from a table.’

On Cloud: In Cloud, resources i.e., compute power or RAM are just click away. In fact, now a days these can automatically increase based on your query requirements. So, if you are using Cloud then, STRICKLY, no one is allowed to run any query either in PROD, UAT or Dev environment without any filter, means in a SQL command one must use WHERE clause as each query is attached with a dollar value. In Cloud, filtering becomes THE FACTOR to maintain resources utilization at minimum, considering as mentioned each resource costs a dollar value.

On-Premises: In contrast of Cloud, resources i.e., compute power or RAM in on-premises systems is limited. To increase resources on premises there is a need for a complete buying cycle by procurement which takes months. With static resources, if someone runs a query without a filter, that query can hold onto all the available resources, locking the rest of the queries. This can create delays in providing other datasets to top management, means delays in decision making, means losing business opportunities, means loss and no one like losses right. In nutshell, in on-premises solutions, filter also is very critical. At the same time developers has the liberty to us any kind of query in development environment as size of data is limited and static.

Data Validation is about verifying or validating the correctness and quality of data against the source. In other words, reconciliation of imported and processed data with its source data.

Data Validation can be done before pulling data and as well after data is processed but must be done before passing data to end users.

Sometime, data changes its shape after transformation from source to target, in this situation Data Validation is conducted against business rules implemented during transformation.

Types of Data Validation

• Constraints Validation checks the accuracy of all types of constraints like primary key, foreign keys, unique, not null, values check and query performance constraint.
• Structured Validation checks the structures of tables and columns.
• Data Range Validation checks the allowable data range to be allowed for processing e.g., if only latest data is allowed to be process the only latest data should get processed.
• Code Validation checks the quality of the syntax. There are ways to write quality and optimized code.
• Data Type Validation checks the type of data to be process e.g., if a data type of a column is Date then only data format with date values should be processed.