Knowledge Hub

Data Silos is always confused with Data Island.

Data Islands are those datastores which are not accessible from external world may be due technology barriers, but Data Silos are those which are only accessible for their relevant stakeholders.

There can be a lot of reasons why data is in silo.

• A particular datastore might be hosting confidential data e.g., finance data, audit data etc.
• A particular data which is not readable e.g., semi or unstructured data sets.
• There is Security or Privacy concerns, so data is not shared among another departments.
• The quality of data is so low that it can’t be shared with management as it may cause wrong decisions and may damage the company’s image.
• Sometime, in development environment different database instances are created for different development teams so if one changes data or table’s structures, other development teams are not impacted.

Why should Data Silos not exist?

There are many ways to achieve data privacy like data generalization, data swapping, data masking, data encryption etc., all these techniques change original data which of course is not desired by most. All these are done to ensure data should not fall into wrong hands.

Data Synthetic is something which will really solve the problem statement like anonymize data before sharing with developers, data science teams for data models etc.

Data Synthetic is about generating new dataset using computer simulations. This dataset contains fake data with the right properties or attributes. For example, customer information is required in almost every application or system, but it can’t be shared to protect individual identity. Data Synthetic tools create a dummy dataset for customers with name, email, contact number, family details etc. but nothing related to original data. This way, developers can build the data pipelines and data scientists can build their data models without thinking of breaching any regulatory or compliance clauses.

There are four types of instructions to pull data i.e., SISD, SIMD, MISD and MIMD.

• SISD: Single Instruction Stream and Single Data Stream
• SIMD: Single Instruction Stream and Multiple Data Stream
• MISD: Multiple Instruction Stream and Single Data Stream
• MIMD: Multiple Instruction Stream and Multiple Data Stream

Before we jump into understanding all above mentioned techniques, let’s first understand what is a Core in a machine or in a computer or in a server etc.?

A CPU Core is built of billions of microscopic transistors to process instructions passed to it. One instruction is handled by One core, if someone needs multitasking or multiple instruction in parallel, minimum 2 cores are required.

We know when we insert or load data in a computer or server or in a node, it gets saved in hard disk. We won’t go in to details how it is stored on the hard disk. We can imagine when we write something on paper, consider the same way, data is saved in a hard disk.

Data Vectorization is something very common nowadays, especially since the inception of Big Data or Hadoop. It’s not like it was not in-use in the past, but I would say it was not famous.

All of us must have heard about MPP (Massive Parallel Processing) right? But do we know how it works at the back end? Data Vectorization is about enabling parallel processing to fetch data.

To understand MPP, let’s refer to four types of instructions to pull data in computer architecture i.e., SISD, SIMD, MISD and MIMD (all explained in separate topic). In this topic, we will be referring to SISD and SIMD only.

Traditionally, a computer or machine or a server works in SISD mode i.e., Single Instruction Stream and Single Data Stream means each instruction fetches required data one by one.

In Data Vectorization, we change our approach to MPP mode i.e., computer or machine or server starts working in SIMD mode i.e., Single Instructure and Multiple Data, means if one query is executed and data is residing in multiple data nodes, data from all nodes will be pulled in parallel making computation must faster as compared to SISD.

Before we understand Data Due Diligence, let’s first have a quick refer what is Due Diligence.

Due diligence is an enquiry, inspect, or assessment executed to check evidence or specifics of a matter beneath attention.

For example, when we are buying something we first check e.g., it is made of which country, it’s a new or used item, it’s produced by which company, what’s the quality of the items etc. All these checks and balances come under due diligence process.

Wikipedia

‘Due diligence is the investigation or exercise of care that a reasonable business or person is normally expected to take before entering into an agreement or contract with another party or an act with a certain standard of care.’

There can be many sub processes involved based on domains, businesses, or solution requirements.

A merger and acquisitions (M&A) process is very old process where due diligence is a must, of course when we’re merging or buying a million-dollar company, we need to make sure our decision is correct.

Data Maturity is the key to becoming a Data Driven Organization. There are many factors to judge it.

‘Maturity refers to stability level of an entity or a process or a product.’

Maturity is not about how many stages of process has passed through, it’s about how stable the product is after implementation.

You must have heard people referring to some individual that he is a mature person so let him take his own decision. Same way, decision can be mature or immature which depend on the result from those decisions.

In this era where Data has become the most powerful input for any organization, well it has been since its start. But now technologies are so MATURE that insight from data analytics are taken very seriously and million-dollar or sometimes billion-dollar deals are taken based on Data Maturity.

Everyone remembers the acquisition of WhatsApp right? Multi-billion-dollar deal just based on how many users is using it and what is the potential growth on a simple mobile application.

As much as Data Filtering is simple as much as it’s so important in Data World.

‘Data Filtering is about reducing the number of rows while pull data from a table.’

On Cloud: In Cloud, resources i.e., compute power or RAM are just click away. In fact, now a days these can automatically increase based on your query requirements. So, if you are using Cloud then, STRICKLY, no one is allowed to run any query either in PROD, UAT or Dev environment without any filter, means in a SQL command one must use WHERE clause as each query is attached with a dollar value. In Cloud, filtering becomes THE FACTOR to maintain resources utilization at minimum, considering as mentioned each resource costs a dollar value.

On-Premises: In contrast of Cloud, resources i.e., compute power or RAM in on-premises systems is limited. To increase resources on premises there is a need for a complete buying cycle by procurement which takes months. With static resources, if someone runs a query without a filter, that query can hold onto all the available resources, locking the rest of the queries. This can create delays in providing other datasets to top management, means delays in decision making, means losing business opportunities, means loss and no one like losses right. In nutshell, in on-premises solutions, filter also is very critical. At the same time developers has the liberty to us any kind of query in development environment as size of data is limited and static.

Data Validation is about verifying or validating the correctness and quality of data against the source. In other words, reconciliation of imported and processed data with its source data.

Data Validation can be done before pulling data and as well after data is processed but must be done before passing data to end users.

Sometime, data changes its shape after transformation from source to target, in this situation Data Validation is conducted against business rules implemented during transformation.

Types of Data Validation

• Constraints Validation checks the accuracy of all types of constraints like primary key, foreign keys, unique, not null, values check and query performance constraint.
• Structured Validation checks the structures of tables and columns.
• Data Range Validation checks the allowable data range to be allowed for processing e.g., if only latest data is allowed to be process the only latest data should get processed.
• Code Validation checks the quality of the syntax. There are ways to write quality and optimized code.
• Data Type Validation checks the type of data to be process e.g., if a data type of a column is Date then only data format with date values should be processed.

Data Inventory is a very simple term but sometime get a little bit confused with Data Catalog and Data Assets etc.

Data Assets is about knowing which kind of data we have in our organization with monetary value.

Data Catalog is about knowing system details, databases, tables, and columns and then mapping those with business requirements.

Data Inventory is about consolidating information from Data Asset and Data Catalog with addition details like.

• where this data comes from
• who generated this data?
• when it was generated
• frequency of inserts, updates, and deletes.
• contents details
• security and privacy approaches
• metadata management approaches
• data quality standards
• transformation details
• business rules
• data models
• list of software
• name of all the applications
• licensing details
• data sources details
• target systems details.
• owners of source and target systems

We all maintain inventory of something or the other, either it’s at home or office. A simple example is Passwords Inventory, now a days there are many social media applications then Hotmail, Gmail, etc. We normally maintain all those passwords somewhere safe right so that is your Password Inventory.

Data Curation is about generating, managing, and maintaining data for future use.

‘Curation comes from the Latin word curae, meaning "care for" -- a meaning it still has today, including in relation to data.’

Data Curation is about taking care of the data from start till end. From extracting till loading. From management to governance. From raw till transformed. From generation till secured and private storage. From open till encryption. As the name says, CARE FOR. Data Curation is also about taking steps to preserve data for future reference as well. Data Curation is the process of preparing data for business and for decision support systems. There is nothing complex about its definition 😊.

Steps for Data Curation

• Understanding the data
• Transformation
• Metadata Management
• Cleansing
• Reconciliation

Stages of Data Curation

• Storing --> Sharing --> Discoverable