Knowledge Hub

Data Dividend is something everyone must know either IT Professional or End User.

We all know what a Dividend is in the Stock Market. It's when a user buys a Stock of a particular company and keeps it for a certain time, the company pays the user a certain amount in Quarterly or Annual basis. The reason is the company is keeping users' money for a particular duration. At the backend, the company is investing that money in its business to generate revenue. Then a certain percentage of that revenue is shared with the user in the shape of a Dividend.

This era is to generate customer data either via mobile applications or via social media portals like Facebook, Twitter, Instagram, Amazon, Alibaba, etc. Users use these applications and, in a way, give their DATA instead of MONEY like their buying/ selling behavior, their search trend, their family/ friends interactions, etc. to above-mentioned companies which then use that DATA for marketing & advertisement, for selling services, to production houses, to industries, to inventory houses, etc. In this cycle, all these companies are making Fortunes, and nothing has been shared with the ENTITY i.e., end-users who are the core participants for the data generation.

Data Assets can be anything that generates data for decision making which ultimately generates revenue for the organization.

We discussed Data Catalog, Data Dictionary and Data Glossary. Data Assets are the perfect Business Case for having the above-mentioned 3 Ds, as One must know what the data is all about and where it is residing. For Data-Driven companies, Data Assets are the most valuable artifacts which can be systems, websites, portals, mobile applications, output excel files, dashboards, reports, campaigns etc.

Facebook, Google, Twitter etc. are the best examples of which use its data as an Asset to run their businesses.

There are multiple ways to maintain Data Assets.

• Cutdown Cost of Data Management by deleting/ purging unnecessary data.
• Make max value out of it by selling it to Generate Revenue
• Manage Data Catalog
• Protection and Security Measures

‘In nutshell, Data Assets are those datasets which has ROI and are able to general revenue.’

Data Liabilities is not handled properly and can severely damage the value of Data Assets.

If Data Liabilities overcome Data Assets in an organization, it can create negative impact of data value.

In-contrast with Data Assets, if Data hosted in your databases, data warehouse, data marts, data lake, data vault, business vault etc. are not utilized properly with no proper ROI and at the same it can’t be used for revenue generation, nor for management decision making then it becomes a liability for your organization. On other words, you are paying for something and not getting anything.

Data Liabilities can also become the reason for Data Loss and failure to ensure proper handling of data usage, data sharing, data quality and data integrity.

One of the major KPI for any CDO should be to reduce data liabilities as much as possible.

Not to mention, hosting and processing data is not a cheap affair plus if management or users can’t utilize it then it’s not worth storing it.

Data Citizens is a very important role in Data field with respect to Data Science activities which still hasn’t been identified as a NORM, but time is not far when just like Data Analysts, Data Citizens will also become one of the MUST-TO-HAVE roles in every department.

‘Today, if 80%+- of Data Science projects are failing it's because of the absence of these Data Citizens roles.’

Data Analysts are functional consultants with skill sets like excel, databases using SQL, reports, and dashboards to dig and present data related to management KPI(s), later these analysis results are passed to ETL/ ELT developers or Data Engineers to implement data pipelines for Productionisation.

Data Citizens are also functional consultants with the machine learning skill sets to do initial data science analysis using DRAG & DROP tools to create, test and process machine learning models to run and predict KPI(s) for business and management, later these models are passed to Data Scientists to implement ML models for Productionisation.

We all have heard about Mean and Median values but not Spread.

Before jumping into Spread, let's have a quick definition of Mean and Median which will assist in understanding Spread.

• Mean: The mean (informally, the “average “) is found by adding all the numbers together and dividing by the number of items in the set: (10 + 10 + 20 + 40 + 70) / 5 = 30.
• Median: The median is found by ordering the set from lowest to highest and finding the exact middle. The median is just the middle number: 20.
• Spread: The simple explanation of Spread is e.g., how variable the data is or how spread out the data values are in the dataset, or it is also referred to as a measure of dispersion etc., from the Mean or Median values.

Let’s try to understand with an example. Let’s say we have a dataset with the customers’ age from 20 to 60.

Breakdown

• 8 customers with Age [10]
• 15 customers with Age [20]
• 12 customers with Age [30]
• 7 customers with Age [40]
• 3 customers with Age [60]

‘Intuition is the ability to understand something without any need for conscious reasoning.’

Intuition is the ability to acquire knowledge without conscious reasoning. It includes but is not limited to direct access to unconscious knowledge, and the ability to understand something instinctively, without any need for conscious reasoning.

Now, how does it relate to Data? I would say, Data Intuition is one of the most important capabilities any data analyst, data citizen, data scientist and management should have. Yes, decisions can be taken based on data, but one should not take a decision, if his/ her intuition is not aligned.

For example, there are sales of 200k drinking bottles per month, expected from a particular area. But suddenly, 500k bottles were sold, NOW your data intuition should shake you about the result. Now you can start asking WHY’s, till you have all the evidence of the sales. It could be a fake result just to show higher sales to get a bonus 😊.

This is again a very crucial topic but is heavily ignored in planning to host data in the big data ecosystem. It can impact the performance of analysis and even data pipelines execution.

Most of us know database tables are used to store data but at the backend where it's stored, we normally don’t bother. As far as databases are concerned, it is still fine to ignore the file format behind it but when it comes to Big Data it’s of utmost importance to plan the right File Formats for the right Data processing.

There are normally two kinds of data usage 1) to do ETL/ ELT 2) to extract data.

• Row File Format: In row format type, when the user requests to extract data from one column, the whole row is extracted in memory and a particular column is displayed on the screen. 
• Columnar File Format: In columnar format type, when a user requests to extract data from one column, only that column data is extracted in memory and displayed on screen.

Query Optimization means to find the fastest route to execute the query. During this process, multiple execution plans are generated and at the end the best one is executed.

Steps involved in this process.

• Query
• Parse & Translator
• Relational Algebra
• Optimizer
• Execution Plan
• Evaluation Engine
• Query Output

The core capability of Query Optimization is to search for an execution plan which can find the least time required for query evaluation.

Phases of Query Processing

• Parsing & Translation
• Optimization
• Evaluation

Example:

We normally submit queries in SQL format. In the first phase of Parsing & Translation, SQL query is translated in an internal form where the parser validates the syntax of the query and the relations of the query in the database and so on. Then it is converted into a relational algebra expression, something like below.

Query:
Select col1 from table1 where col1=’XYZ’

Relational Algebra Expression:

An Index is used to enhance the performance of database queries on tables. We can create as many indexes as possible we need or want on a particular table. Please note, in contrast, having more than required indexes can decrease performance as well.

Normally, an index is created when the database starts growing.

Why are indexes needed?

A famous example for better understanding; imagine walking into the Library of Congress and be given the task of finding a specific publication within 10 minutes. Would you be able to complete this task within the given time frame? The Library of Congress is considered the largest library in the world and it houses approximately 170 million items. Now, if you are a smart chap like me 😊, the first thing we would do is ask for access to the library’s index (which can be manual list or a list in a computer system but a list of names and locations of each book in the library) because indexes contain all the necessary information needed to access items quickly and efficiently. In the same manner, a database index contains all the necessary information to access data quickly and efficiently.

Partitioning is a data organization scheme in which table data is divided across multiple storage objects called data partitions according to values, in one or more table columns. Each data partition is stored separately. It consists of its data, indexes, configuration files, and transaction logs. For example, there is a Salary table with 10 years of data. It can be partitioned yearly. At the backend, at the storage level, this table will be divided into 10 smaller datasets i.e.,

1 for each year. Keynote here is, all partitions will be stored wherever the empty slots are, maybe 4 partitions including years 2 & 3 got stored in 1 node of 10 nodes-cluster. This is called Shared-Architecture. Now, a query for years 2 & 3 will go and hit only two partitions making it much faster but as both partitions were on 1 node so one node processing is used.

Partitioning Techniques

• Single-level Partitioning
  • Hash Partitioning
  • Range Partitioning
  • List Partitioning
• Composite Partitioning
  • Range–Range Partitioning
  • Range–Hash Partitioning
  • Range–List Partitioning
  • List–Range Partitioning
  • List–Hash Partitioning
  • List–List Partitioning