Knowledge Hub

Obvious question, difference between Hadoop, HDFS and Hive?

Hadoop is an Open-Source Framework to store structured, semi/unstructured data. Default block size of 128MB (changeable).

HDFS is a Hadoop Files System just like FAT, NTFS, EXT etc.

Hive is only a wrapper on top of files stored in HDFS. Hive behaves just like tables in traditional databases, but it doesn't store any data, it just manages table structures on top of HDFS files. There are 2 ways Hive works, either store files in HDFS directly or create tables which will create HDFS files at the back end. The difference in these approaches is 1) each file stored in HDFS creates its own block so if there are 2 small files e.g., 50 MB each, then there will be 2 blocks created. 2) on the other hand, Hive manages its own blocks so if there are 2 small files of e.g., 50 MB each, then Hive will store both in a single block.

Consider 1000s of 1MB files float in from social media means 1000s of 1MB small blocks created impacting MPP performance. The best practice is to have blocks of 128MB.

It refers to the staggering amount and variety of data produced by businesses every day. This is largely due to the variety of enterprise software, mobile apps, storage systems, and data formats each company relies on.

We have data in the shape of Video, Audio, Email, PDF, JSON, XML, Text etc. In other words, data is scattered in different formats and unless all formats are not converted in a standardized format, extracting information will also be challenging or time-consuming.

Imagine that you need to complete your taxes, but all your relevant papers are in different drawers, hidden in closets, and stuffed under couch cushions. Now imagine that you have multiple copies of the forms in these places, and some are written in Greek, while others are written in English and Spanish. How will you do your taxes, or clean your house for that matter, when this is the state of things? Unfortunately, this is a problem that is starting to plague companies across the world. This is Data Sprawl.

Dormant Data and Dark Data are mostly confused with each other.

Dark Data

‘Data that is not utilized is Dark Data.’

Dark Data is data that is not in use due to lack of technology and is not a Data Asset. Dormant Data is the data, which is useable, but decision-makers don't use it, or it’s not required for analysis but that doesn't mean you can delete it or purge it, so we keep it in Tape drives, in Hadoop and nowadays Object Storage is also good choice.

Before the introduction of Big Data, the % age of Dark Data was around 90% as it included semi and unstructured datasets and sometimes structured as well. For example, JSON or XML files, Video or Audio files, Logs, Metadata etc., all these datasets were there but technologies were either not there or the affair to analyze those were very expensive. The only reason companies kept this kind of data was for Compliance or Audit purposes. Since the Big Data ecosystem has come onto the surface and hardware has become dead cheap so hosting and processing these datasets has become possible.

As we discussed in topic i.e., Dark Data, that it is utmost important for a CDO (Chief Data Officer) to minimize Dark Data as much as possible, in the organization. In other words, the more the Dark Data, the more the cost without any ROI. Time is not far when Dark Data Management will become the most essential and obvious KPI for CDO (Chief Data Officer).

Now the questions are.

• Is all Dark Data bad? NO.
• Is all Dark Data delete able or purge able? Yes and No.

‘Data Detritus means that data, which is no more required, which is of no use anymore etc.’

• Dark data is something which was and is, not in-use.
• Dormant data is something which is hardly in-use.
• Detritus data is something which was in-use but will not be in-use, anymore.

In other words, Detritus data was once valuable before it became Dormant, as compared to traditional Dark Data (explained separately) which was never in-use due to technologies barriers etc. Detritus data is normally not deleted or purged as it can still be required for Regulatory or Compliance parties.

Data Dividend is something everyone must know either IT Professional or End User.

We all know what a Dividend is in the Stock Market. It's when a user buys a Stock of a particular company and keeps it for a certain time, the company pays the user a certain amount in Quarterly or Annual basis. The reason is the company is keeping users' money for a particular duration. At the backend, the company is investing that money in its business to generate revenue. Then a certain percentage of that revenue is shared with the user in the shape of a Dividend.

This era is to generate customer data either via mobile applications or via social media portals like Facebook, Twitter, Instagram, Amazon, Alibaba, etc. Users use these applications and, in a way, give their DATA instead of MONEY like their buying/ selling behavior, their search trend, their family/ friends interactions, etc. to above-mentioned companies which then use that DATA for marketing & advertisement, for selling services, to production houses, to industries, to inventory houses, etc. In this cycle, all these companies are making Fortunes, and nothing has been shared with the ENTITY i.e., end-users who are the core participants for the data generation.

Data Assets can be anything that generates data for decision making which ultimately generates revenue for the organization.

We discussed Data Catalog, Data Dictionary and Data Glossary. Data Assets are the perfect Business Case for having the above-mentioned 3 Ds, as One must know what the data is all about and where it is residing. For Data-Driven companies, Data Assets are the most valuable artifacts which can be systems, websites, portals, mobile applications, output excel files, dashboards, reports, campaigns etc.

Facebook, Google, Twitter etc. are the best examples of which use its data as an Asset to run their businesses.

There are multiple ways to maintain Data Assets.

• Cutdown Cost of Data Management by deleting/ purging unnecessary data.
• Make max value out of it by selling it to Generate Revenue
• Manage Data Catalog
• Protection and Security Measures

‘In nutshell, Data Assets are those datasets which has ROI and are able to general revenue.’

Data Liabilities is not handled properly and can severely damage the value of Data Assets.

If Data Liabilities overcome Data Assets in an organization, it can create negative impact of data value.

In-contrast with Data Assets, if Data hosted in your databases, data warehouse, data marts, data lake, data vault, business vault etc. are not utilized properly with no proper ROI and at the same it can’t be used for revenue generation, nor for management decision making then it becomes a liability for your organization. On other words, you are paying for something and not getting anything.

Data Liabilities can also become the reason for Data Loss and failure to ensure proper handling of data usage, data sharing, data quality and data integrity.

One of the major KPI for any CDO should be to reduce data liabilities as much as possible.

Not to mention, hosting and processing data is not a cheap affair plus if management or users can’t utilize it then it’s not worth storing it.

Data Citizens is a very important role in Data field with respect to Data Science activities which still hasn’t been identified as a NORM, but time is not far when just like Data Analysts, Data Citizens will also become one of the MUST-TO-HAVE roles in every department.

‘Today, if 80%+- of Data Science projects are failing it's because of the absence of these Data Citizens roles.’

Data Analysts are functional consultants with skill sets like excel, databases using SQL, reports, and dashboards to dig and present data related to management KPI(s), later these analysis results are passed to ETL/ ELT developers or Data Engineers to implement data pipelines for Productionisation.

Data Citizens are also functional consultants with the machine learning skill sets to do initial data science analysis using DRAG & DROP tools to create, test and process machine learning models to run and predict KPI(s) for business and management, later these models are passed to Data Scientists to implement ML models for Productionisation.

We all have heard about Mean and Median values but not Spread.

Before jumping into Spread, let's have a quick definition of Mean and Median which will assist in understanding Spread.

• Mean: The mean (informally, the “average “) is found by adding all the numbers together and dividing by the number of items in the set: (10 + 10 + 20 + 40 + 70) / 5 = 30.
• Median: The median is found by ordering the set from lowest to highest and finding the exact middle. The median is just the middle number: 20.
• Spread: The simple explanation of Spread is e.g., how variable the data is or how spread out the data values are in the dataset, or it is also referred to as a measure of dispersion etc., from the Mean or Median values.

Let’s try to understand with an example. Let’s say we have a dataset with the customers’ age from 20 to 60.

Breakdown

• 8 customers with Age [10]
• 15 customers with Age [20]
• 12 customers with Age [30]
• 7 customers with Age [40]
• 3 customers with Age [60]

‘Intuition is the ability to understand something without any need for conscious reasoning.’

Intuition is the ability to acquire knowledge without conscious reasoning. It includes but is not limited to direct access to unconscious knowledge, and the ability to understand something instinctively, without any need for conscious reasoning.

Now, how does it relate to Data? I would say, Data Intuition is one of the most important capabilities any data analyst, data citizen, data scientist and management should have. Yes, decisions can be taken based on data, but one should not take a decision, if his/ her intuition is not aligned.

For example, there are sales of 200k drinking bottles per month, expected from a particular area. But suddenly, 500k bottles were sold, NOW your data intuition should shake you about the result. Now you can start asking WHY’s, till you have all the evidence of the sales. It could be a fake result just to show higher sales to get a bonus 😊.