Knowledge Hub

We have covered Business Intelligence, Business Analytics, Data Modelling and Data Cardinality, it's the right time to discuss Data Visualization.

Data Visualization is what is seen by End Users. The Success or Failure of the project is determined based on it. As the phrase goes, 'If you can't draw it, you don't Know it.' And it’s very true. One can spend millions of dollars, can implement a perfect Data Strategy, can have perfect EDW or Data Lake etc. etc. but after all the hard work if the End User can't see what is implementation, if the End User is unable to make use of the data for DSS, if End User will have to go back to their Finance or Technical departments for manual data extraction then all the efforts put-in is nothing more than Waste of Time and Dollars.

Now, there are rules to have good Data Visualization which is relevant to any visualization tools in the market like Tableau, Qlik, Power BI, OBIEE, Cognos and many more.

Heat Map is a graphical representation of objects at any specific location. Nowadays, it is used to identify highly populated area(s) during Covid-19 era. Heat maps make it easy to visualize complex data and understand it briefly.

Now, the question is how Heat Map software knows where the location is. When we open any webpage or any app like Facebook or Twitter etc., via mobile or laptop, our location is picked up. That location is based on Lat and Long (Latitude and Longitude) values. These values are then sent to databases which are then picked up by the Heat Map software for graphical presentation.

• Latitude is the measurement of distance north or south of the Equator. It is measured with 180 imaginary lines that form circles around the Earth east-west, parallel to the Equator.
• Longitude is the measurement east or west of the prime meridian. Longitude is measured by imaginary lines that run around the Earth vertically (up and down) and meet at the North and South Poles. These lines are known as meridians. Each meridian measures one arc degree of longitude. The distance around the Earth measures 360 degrees.

DIKW model

‘Data --> Information --> Knowledge --> Wisdom’

Generally, Data, Information, Knowledge, and Wisdom are arranged in the above sequence. How about considering another angle and trying to arrange these 4 terms in different sequence?

For me Information and Knowledge both comes from Data and Wisdom comes from both Information and Knowledge.

Difference between Data and Information?

As we all know, now a days, Data is everywhere either it is generating from System/ Mobile Applications, Online Websites/ Portals, Videos, Audios, logs etc.

• Is all Data equal to Information? No, it’s NOT.
• Can all Data become Information? Yes, it CAN, based on the nature of the business or requirement.

To answer above, one must first understand what Information is. Information is tightly coupled with what one wants to know. You must have heard this question on many occasions or events or counters etc. like What Information do you need?

This is also a very confusing term in Data domain, a lot of people are not sure how to assure that everyone follows Ethics in their day-to-day activities while interacting with Data. Just like CDE (Critical Data Element), Data Ethics is also a principle and must be managed at organization level and followed at individual level.

Now, the big question is which data falls under the ambarella where Data Ethics must be followed. There are multiple factors involved and everyone has his/ her own level of ethics to handle based on their roles and responsibilities.

Data Ethics is also a term which if not taken care of properly, can damage a company’s image with the general public. To make sure organizations follow ethics, different regulatory bodies in different parts of the world and in different sectors give guiding principles to follow. It doesn’t stop here, if the organization doesn’t end up following all the measures asked-for by those regulatory bodies then they can be penalized with heavy amounts as penalties.

Data Surfing is always confused with Data Analysis. There is a very thin line between Data Analysts and Data Surfers.

Data Surfing is normally done by non-technical people like management who doesn’t know how to write any kind of code to access data at database level.

Data Surfers normally stays at the level of Dashboards and Reports where they just click here and there to explore data for required information.

Business Intelligence (BI) tools came into being to cater Data Surfers World where data was required to be presented for non-technical people who could view data for decision making.

Era(s) for Data Surfers

• Static Reports: In the beginning, business users used to share their data requirements with IT who then used to create dashboards and reports. Those dashboards/ reports and data was not changeable, it was in paper format or in excel format or in text format etc., means after Go-Live if business want one more column or if latest data was required then the whole development cycle had to restart from scratch.
• Dynamic Reports: In the second era, BI tools came-in to cater for this gap. To start with, dashboards/ reports were still not changeable, but data was directly connected with backend databases, so data was always latest at front-end. This was a huge jump, as now, to view new data developer don’t have to recreate front-end again and again.
• Adhoc Reports: Latest era is where dashboards/ reports are also changeable. Now, end users and management can drag and drop tables and columns in dashboards/ reports.

Data Island has kind of disappeared as integration technologies have matured.

‘Data Island is a datastore(s) which is/ are not accessible.’

In previous eras when network security was a big concern or even before, when network connectivity was not even there, the datastores were considered as Data Island as external integration was not possible.

Let’s try to understand with examples. We all know WIFI has become a norm in our lives and living without it is like living without Air right? In past when we used to visit a friend’s place we used to greet first, now we first ask, ‘What’s your WIFI Password’ lolz 😊?’. But if you are as old as I am, before 2005 WIFI was not a thing. There were ONLY cables which used to connect our desktops or laptops with network, well I still prefer cable. During that era consider a datastore sitting in different city or region then how to connect? In that situation, we had floppy drives for small data transfer (yeah, I won’t explain what is floppy drive lolz 😊, old folks know what I am talking about. Let’s let newbies find out what is floppy drive), and for huge data we used to have big hard disks and sometimes desktops itself were moved to move data from one location to another location. This datastore, which was not connectable with other datastore, was call Data Island.

Data Silos is always confused with Data Island.

Data Islands are those datastores which are not accessible from external world may be due technology barriers, but Data Silos are those which are only accessible for their relevant stakeholders.

There can be a lot of reasons why data is in silo.

• A particular datastore might be hosting confidential data e.g., finance data, audit data etc.
• A particular data which is not readable e.g., semi or unstructured data sets.
• There is Security or Privacy concerns, so data is not shared among another departments.
• The quality of data is so low that it can’t be shared with management as it may cause wrong decisions and may damage the company’s image.
• Sometime, in development environment different database instances are created for different development teams so if one changes data or table’s structures, other development teams are not impacted.

Why should Data Silos not exist?

As much as Data Filtering is simple as much as it’s so important in Data World.

‘Data Filtering is about reducing the number of rows while pull data from a table.’

On Cloud: In Cloud, resources i.e., compute power or RAM are just click away. In fact, now a days these can automatically increase based on your query requirements. So, if you are using Cloud then, STRICKLY, no one is allowed to run any query either in PROD, UAT or Dev environment without any filter, means in a SQL command one must use WHERE clause as each query is attached with a dollar value. In Cloud, filtering becomes THE FACTOR to maintain resources utilization at minimum, considering as mentioned each resource costs a dollar value.

On-Premises: In contrast of Cloud, resources i.e., compute power or RAM in on-premises systems is limited. To increase resources on premises there is a need for a complete buying cycle by procurement which takes months. With static resources, if someone runs a query without a filter, that query can hold onto all the available resources, locking the rest of the queries. This can create delays in providing other datasets to top management, means delays in decision making, means losing business opportunities, means loss and no one like losses right. In nutshell, in on-premises solutions, filter also is very critical. At the same time developers has the liberty to us any kind of query in development environment as size of data is limited and static.

Data Validation is about verifying or validating the correctness and quality of data against the source. In other words, reconciliation of imported and processed data with its source data.

Data Validation can be done before pulling data and as well after data is processed but must be done before passing data to end users.

Sometime, data changes its shape after transformation from source to target, in this situation Data Validation is conducted against business rules implemented during transformation.

Types of Data Validation

• Constraints Validation checks the accuracy of all types of constraints like primary key, foreign keys, unique, not null, values check and query performance constraint.
• Structured Validation checks the structures of tables and columns.
• Data Range Validation checks the allowable data range to be allowed for processing e.g., if only latest data is allowed to be process the only latest data should get processed.
• Code Validation checks the quality of the syntax. There are ways to write quality and optimized code.
• Data Type Validation checks the type of data to be process e.g., if a data type of a column is Date then only data format with date values should be processed.

As the name says all, Data Sharing is about making data accessible to all the relevant stakeholders.

‘Data Sharing is about eliminating Dark Data.’

Data Sharing is one of the key components in Data Driven Organization where organizations aim to build data culture with Data Liberalization.

Data Sharing Channels

• Using Business Intelligence tools
• Using languages like SQL, Python etc.
• Using APIs or direction Using JDBC/ ODBC Connectors
• Using built-in Drivers
• Using Machine Learning or Using Data Mining Models
• Exporting in Excel, CSV, FF formats
• Pushing data to other databases and applications
• Letting other databases and application to Pull data.
• ETL/ ELT tools
• Audit, Compliance or Regulatory Reports etc.