Knowledge Hub

Dormant Data and Dark Data are mostly confused with each other.

Dark Data

‘Data that is not utilized is Dark Data.’

Dark Data is data that is not in use due to lack of technology and is not a Data Asset. Dormant Data is the data, which is useable, but decision-makers don't use it, or it’s not required for analysis but that doesn't mean you can delete it or purge it, so we keep it in Tape drives, in Hadoop and nowadays Object Storage is also good choice.

Before the introduction of Big Data, the % age of Dark Data was around 90% as it included semi and unstructured datasets and sometimes structured as well. For example, JSON or XML files, Video or Audio files, Logs, Metadata etc., all these datasets were there but technologies were either not there or the affair to analyze those were very expensive. The only reason companies kept this kind of data was for Compliance or Audit purposes. Since the Big Data ecosystem has come onto the surface and hardware has become dead cheap so hosting and processing these datasets has become possible.

As we discussed in topic i.e., Dark Data, that it is utmost important for a CDO (Chief Data Officer) to minimize Dark Data as much as possible, in the organization. In other words, the more the Dark Data, the more the cost without any ROI. Time is not far when Dark Data Management will become the most essential and obvious KPI for CDO (Chief Data Officer).

Now the questions are.

• Is all Dark Data bad? NO.
• Is all Dark Data delete able or purge able? Yes and No.

‘Data Detritus means that data, which is no more required, which is of no use anymore etc.’

• Dark data is something which was and is, not in-use.
• Dormant data is something which is hardly in-use.
• Detritus data is something which was in-use but will not be in-use, anymore.

In other words, Detritus data was once valuable before it became Dormant, as compared to traditional Dark Data (explained separately) which was never in-use due to technologies barriers etc. Detritus data is normally not deleted or purged as it can still be required for Regulatory or Compliance parties.

Data Dividend is something everyone must know either IT Professional or End User.

We all know what a Dividend is in the Stock Market. It's when a user buys a Stock of a particular company and keeps it for a certain time, the company pays the user a certain amount in Quarterly or Annual basis. The reason is the company is keeping users' money for a particular duration. At the backend, the company is investing that money in its business to generate revenue. Then a certain percentage of that revenue is shared with the user in the shape of a Dividend.

This era is to generate customer data either via mobile applications or via social media portals like Facebook, Twitter, Instagram, Amazon, Alibaba, etc. Users use these applications and, in a way, give their DATA instead of MONEY like their buying/ selling behavior, their search trend, their family/ friends interactions, etc. to above-mentioned companies which then use that DATA for marketing & advertisement, for selling services, to production houses, to industries, to inventory houses, etc. In this cycle, all these companies are making Fortunes, and nothing has been shared with the ENTITY i.e., end-users who are the core participants for the data generation.

Data Assets can be anything that generates data for decision making which ultimately generates revenue for the organization.

We discussed Data Catalog, Data Dictionary and Data Glossary. Data Assets are the perfect Business Case for having the above-mentioned 3 Ds, as One must know what the data is all about and where it is residing. For Data-Driven companies, Data Assets are the most valuable artifacts which can be systems, websites, portals, mobile applications, output excel files, dashboards, reports, campaigns etc.

Facebook, Google, Twitter etc. are the best examples of which use its data as an Asset to run their businesses.

There are multiple ways to maintain Data Assets.

• Cutdown Cost of Data Management by deleting/ purging unnecessary data.
• Make max value out of it by selling it to Generate Revenue
• Manage Data Catalog
• Protection and Security Measures

‘In nutshell, Data Assets are those datasets which has ROI and are able to general revenue.’

Data Liabilities is not handled properly and can severely damage the value of Data Assets.

If Data Liabilities overcome Data Assets in an organization, it can create negative impact of data value.

In-contrast with Data Assets, if Data hosted in your databases, data warehouse, data marts, data lake, data vault, business vault etc. are not utilized properly with no proper ROI and at the same it can’t be used for revenue generation, nor for management decision making then it becomes a liability for your organization. On other words, you are paying for something and not getting anything.

Data Liabilities can also become the reason for Data Loss and failure to ensure proper handling of data usage, data sharing, data quality and data integrity.

One of the major KPI for any CDO should be to reduce data liabilities as much as possible.

Not to mention, hosting and processing data is not a cheap affair plus if management or users can’t utilize it then it’s not worth storing it.

Data Citizens is a very important role in Data field with respect to Data Science activities which still hasn’t been identified as a NORM, but time is not far when just like Data Analysts, Data Citizens will also become one of the MUST-TO-HAVE roles in every department.

‘Today, if 80%+- of Data Science projects are failing it's because of the absence of these Data Citizens roles.’

Data Analysts are functional consultants with skill sets like excel, databases using SQL, reports, and dashboards to dig and present data related to management KPI(s), later these analysis results are passed to ETL/ ELT developers or Data Engineers to implement data pipelines for Productionisation.

Data Citizens are also functional consultants with the machine learning skill sets to do initial data science analysis using DRAG & DROP tools to create, test and process machine learning models to run and predict KPI(s) for business and management, later these models are passed to Data Scientists to implement ML models for Productionisation.

We all have heard about Mean and Median values but not Spread.

Before jumping into Spread, let's have a quick definition of Mean and Median which will assist in understanding Spread.

• Mean: The mean (informally, the “average “) is found by adding all the numbers together and dividing by the number of items in the set: (10 + 10 + 20 + 40 + 70) / 5 = 30.
• Median: The median is found by ordering the set from lowest to highest and finding the exact middle. The median is just the middle number: 20.
• Spread: The simple explanation of Spread is e.g., how variable the data is or how spread out the data values are in the dataset, or it is also referred to as a measure of dispersion etc., from the Mean or Median values.

Let’s try to understand with an example. Let’s say we have a dataset with the customers’ age from 20 to 60.

Breakdown

• 8 customers with Age [10]
• 15 customers with Age [20]
• 12 customers with Age [30]
• 7 customers with Age [40]
• 3 customers with Age [60]

‘Intuition is the ability to understand something without any need for conscious reasoning.’

Intuition is the ability to acquire knowledge without conscious reasoning. It includes but is not limited to direct access to unconscious knowledge, and the ability to understand something instinctively, without any need for conscious reasoning.

Now, how does it relate to Data? I would say, Data Intuition is one of the most important capabilities any data analyst, data citizen, data scientist and management should have. Yes, decisions can be taken based on data, but one should not take a decision, if his/ her intuition is not aligned.

For example, there are sales of 200k drinking bottles per month, expected from a particular area. But suddenly, 500k bottles were sold, NOW your data intuition should shake you about the result. Now you can start asking WHY’s, till you have all the evidence of the sales. It could be a fake result just to show higher sales to get a bonus 😊.

This is again a very crucial topic but is heavily ignored in planning to host data in the big data ecosystem. It can impact the performance of analysis and even data pipelines execution.

Most of us know database tables are used to store data but at the backend where it's stored, we normally don’t bother. As far as databases are concerned, it is still fine to ignore the file format behind it but when it comes to Big Data it’s of utmost importance to plan the right File Formats for the right Data processing.

There are normally two kinds of data usage 1) to do ETL/ ELT 2) to extract data.

• Row File Format: In row format type, when the user requests to extract data from one column, the whole row is extracted in memory and a particular column is displayed on the screen. 
• Columnar File Format: In columnar format type, when a user requests to extract data from one column, only that column data is extracted in memory and displayed on screen.

Query Optimization means to find the fastest route to execute the query. During this process, multiple execution plans are generated and at the end the best one is executed.

Steps involved in this process.

• Query
• Parse & Translator
• Relational Algebra
• Optimizer
• Execution Plan
• Evaluation Engine
• Query Output

The core capability of Query Optimization is to search for an execution plan which can find the least time required for query evaluation.

Phases of Query Processing

• Parsing & Translation
• Optimization
• Evaluation

Example:

We normally submit queries in SQL format. In the first phase of Parsing & Translation, SQL query is translated in an internal form where the parser validates the syntax of the query and the relations of the query in the database and so on. Then it is converted into a relational algebra expression, something like below.

Query:
Select col1 from table1 where col1=’XYZ’

Relational Algebra Expression: