Knowledge Hub

By now, we all know what Semantic Layer is, IT defines data elements for business ease. Then there came the era for Business folks defines data elements themselves based on their own understanding and now the latest approach is going towards Machine Learning Models define data element based on open-ended requirements.

A major difference between first two approaches vs Augmented Analytics is, first two were derived by humans either by IT or Business folks so there is always a factor of human era and at the same time human can be biased base on knowledge and information they possess about the business domain. On the other hand, Augmented Analytics is based on automated Machine Learning which learns the data elements while new data sets keep coming in.

The first approach was static as requirements were always coming from business and implemented by IT folks, so any change had a round robin effect.

There are many ways to achieve data privacy like data generalization, data swapping, data masking, data encryption etc., all these techniques change original data which of course is not desired by most. All these are done to ensure data should not fall into wrong hands.

Data Synthetic is something which will really solve the problem statement like anonymize data before sharing with developers, data science teams for data models etc.

Data Synthetic is about generating new dataset using computer simulations. This dataset contains fake data with the right properties or attributes. For example, customer information is required in almost every application or system, but it can’t be shared to protect individual identity. Data Synthetic tools create a dummy dataset for customers with name, email, contact number, family details etc. but nothing related to original data. This way, developers can build the data pipelines and data scientists can build their data models without thinking of breaching any regulatory or compliance clauses.

Data Vectorization is something very common nowadays, especially since the inception of Big Data or Hadoop. It’s not like it was not in-use in the past, but I would say it was not famous.

All of us must have heard about MPP (Massive Parallel Processing) right? But do we know how it works at the back end? Data Vectorization is about enabling parallel processing to fetch data.

To understand MPP, let’s refer to four types of instructions to pull data in computer architecture i.e., SISD, SIMD, MISD and MIMD (all explained in separate topic). In this topic, we will be referring to SISD and SIMD only.

Traditionally, a computer or machine or a server works in SISD mode i.e., Single Instruction Stream and Single Data Stream means each instruction fetches required data one by one.

In Data Vectorization, we change our approach to MPP mode i.e., computer or machine or server starts working in SIMD mode i.e., Single Instructure and Multiple Data, means if one query is executed and data is residing in multiple data nodes, data from all nodes will be pulled in parallel making computation must faster as compared to SISD.

Before we understand Data Due Diligence, let’s first have a quick refer what is Due Diligence.

Due diligence is an enquiry, inspect, or assessment executed to check evidence or specifics of a matter beneath attention.

For example, when we are buying something we first check e.g., it is made of which country, it’s a new or used item, it’s produced by which company, what’s the quality of the items etc. All these checks and balances come under due diligence process.

Wikipedia

‘Due diligence is the investigation or exercise of care that a reasonable business or person is normally expected to take before entering into an agreement or contract with another party or an act with a certain standard of care.’

There can be many sub processes involved based on domains, businesses, or solution requirements.

A merger and acquisitions (M&A) process is very old process where due diligence is a must, of course when we’re merging or buying a million-dollar company, we need to make sure our decision is correct.

Data Maturity is the key to becoming a Data Driven Organization. There are many factors to judge it.

‘Maturity refers to stability level of an entity or a process or a product.’

Maturity is not about how many stages of process has passed through, it’s about how stable the product is after implementation.

You must have heard people referring to some individual that he is a mature person so let him take his own decision. Same way, decision can be mature or immature which depend on the result from those decisions.

In this era where Data has become the most powerful input for any organization, well it has been since its start. But now technologies are so MATURE that insight from data analytics are taken very seriously and million-dollar or sometimes billion-dollar deals are taken based on Data Maturity.

Everyone remembers the acquisition of WhatsApp right? Multi-billion-dollar deal just based on how many users is using it and what is the potential growth on a simple mobile application.

Data Inventory is a very simple term but sometime get a little bit confused with Data Catalog and Data Assets etc.

Data Assets is about knowing which kind of data we have in our organization with monetary value.

Data Catalog is about knowing system details, databases, tables, and columns and then mapping those with business requirements.

Data Inventory is about consolidating information from Data Asset and Data Catalog with addition details like.

• where this data comes from
• who generated this data?
• when it was generated
• frequency of inserts, updates, and deletes.
• contents details
• security and privacy approaches
• metadata management approaches
• data quality standards
• transformation details
• business rules
• data models
• list of software
• name of all the applications
• licensing details
• data sources details
• target systems details.
• owners of source and target systems

We all maintain inventory of something or the other, either it’s at home or office. A simple example is Passwords Inventory, now a days there are many social media applications then Hotmail, Gmail, etc. We normally maintain all those passwords somewhere safe right so that is your Password Inventory.

Data Curation is about generating, managing, and maintaining data for future use.

‘Curation comes from the Latin word curae, meaning "care for" -- a meaning it still has today, including in relation to data.’

Data Curation is about taking care of the data from start till end. From extracting till loading. From management to governance. From raw till transformed. From generation till secured and private storage. From open till encryption. As the name says, CARE FOR. Data Curation is also about taking steps to preserve data for future reference as well. Data Curation is the process of preparing data for business and for decision support systems. There is nothing complex about its definition 😊.

Steps for Data Curation

• Understanding the data
• Transformation
• Metadata Management
• Cleansing
• Reconciliation

Stages of Data Curation

• Storing --> Sharing --> Discoverable

Syndication is also known as synchronization, yes now it must be making sense to you 😊.

‘Data Syndication is a process not a technology.’

The word syndicate comes from the French word syndicate which means "trade union" (syndic meaning "administrator"), from the Latin word syndicus which in turn comes from the Greek word σύνδικος (syndikos), which means "caretaker of an issue"; compare to ombudsman or representative.

‘Data Syndication is to collect and prepare update-to-date dataset for a particular activity.’

For example, prepare a dataset from multiple source systems for the marketing team to plan their upcoming campaigning exercise.

Data Syndication process

• Can handle any type of format as input and output as well.
• It can send alerts.
• It can configure access matrix.
• It provides monitoring dashboards and reports for compliance and audit trials.
• As it’s a process, any expensive or cheap technology can be utilized.
• Output formats can be designed based on customer requirements.
• Can send email, SMS, spreadsheet, database, ML models etc.

Data Supply Chain is not a data capability rather a process.

Just like generic supply chain, Data Supply Chain works in the same manner where it starts from Source Data Stage 🡪 Transformation and Integration Stage 🡪 Consumption Stage

• Source Data Stage is the first stage where raw data is planned and identified i.e., OLTP systems.
• Transformation and Integration Stage is where data is converted based on business requirement in-shape of Data Lake Systems or Data Warehouses
• Consumption Stage is where data is store for business use in-shape of Data Lake Systems or Data Warehouses (OLAP systems) or Semantic Layer etc.

Data Supply Chain is a very critical operational process to manage Data Asset and Liabilities. In the absence of visibility on Data Supply Chain, organization cannot plan ROI as well as TTM which is critical for the Data Monetization.

Note: Data Assets, Data Liabilities and Data Monetization all are explained in separate topics.

Data Lifecycle Management (DLM) is mostly confused with Data Supply Chain which is not 100% correct.

Data Lifecycle Management is about the process from where data was generated till it was removed from the systems whereas Data Supply Chain is about where data come from and where it goes.

There are many sets of steps for Data Lifecycle management, we will be looking into the most commonly used.

Steps in Data Lifecycle Management (DLM)

• Data Generation means the channel where data is getting generated e.g., mobile application, CRM, ERP, HRM, CCTV, Social-Media, Audio(s), Emails, Word document, PDF, Images, Scans etc.
• Data Processing means, what are the business rules based on which raw data is transformed in required format which can be via ETL or ELT or via Data Mining or Data Science Machine Learning Modelling etc.
• Data Reconciliation, Presentation and Utilization means how transformed data is matched with raw data then how it’s been shown on Reports and Dashboard, and lastly for what purpose it has been used.
• Data Storage Lifespan and Extraction means, for how long the data will be kept and what are the means to extract it which can be manual or can be via automated tools.
• Data Removal means how data has been removed and based on which criteria(s).