Knowledge Hub

Data Virtualization, Data Federation and Data Consolidation all are discussed together with respect to exposing data to data consumers.

Where Data Virtualization and Data Federation do not host any copy of data in its environments, Data Consolidation on the other hand keeps complete historical copies of all the data coming from all the source systems. In this process sometimes, a complex ETL or ELT takes place to process data in different data models and sometimes one to one data model is maintained.

Yes, I am referring to Data Warehouses, Data Marts, Data Lakes, Delta Lakes, Data Vault, Business Vaults etc., all these data stores fall under the umbrella of Data Consolidation environments.

In the current era there are many data stores like Database, Data Warehouse, Data Marts, Big Data, Data Lake, Data Vault, Business Vault, SQL, and NoSQL etc. Use of all these terms is different with their own horizon, their own skill sets whereas tools and languages can be the same for all of these.

First one must understand is that Database is a datastore, we can’t link it to any data modelling solution. In other words, OLTP or Data Warehouse or Data Mart or OLAP all use databases as a medium to store their datasets. Additionally, Databases can be SQL or NoSQL.

Now, OLTP is on one side of the line and Data Warehouse/ Data Marts and Big Data/ Data Lake etc. are on the other side. Normally, databases are called OLTP (Online Transactional Processing) whereas Data Warehouse/ Data Marts fall under the term OLAP (Online Analytical Processing).

As far as Big Data/ Data Lake is concerned, Data World has yet to tag these with either OLTP or OLAP but for sure these are not meant for OLTP. At the same time NoSQL is stepping into OLTP zone and is supporting OLTP as well.

The idea of a Data Warehouse emerged when organizations realized that transactional systems (OLTP), while excellent for recording day-to-day operations, were never designed for heavy reporting or analytical workloads. As business leaders demanded consolidated views of performance, finance, sales, and customer behaviour, data modelers introduced the concept of a centralized analytical datastore, the Data Warehouse, to relieve OLTP systems from the burden of reporting and to provide trusted, historical insight.

It is important to clarify a common source of confusion: a Data Warehouse is not the same as Dimensional Modelling. The warehouse refers to the central repository that consolidates data from multiple source systems, while dimensional modelling is only one possible technique for structuring that repository. A Data Warehouse can be designed in different ways, depending on organizational priorities such as performance, scalability, or agility.

The term Big Data emerged in the early 2000s as enterprises began facing unprecedented growth in the amount, speed, and diversity of information being generated. Traditional databases and warehouses, while reliable for structured data, were not designed to handle this new reality. Big Data describes not only the massive scale of data itself but also the ecosystem of technologies, methods, and practices developed to capture, store, and analyze it.

At the heart of Big Data is the concept of the “three Vs”, later extended to four or more:

• Volume: the enormous scale of data generated from transactions, machines, sensors, mobile devices, and social platforms.
• Velocity: the speed at which data arrives, often requiring real-time or near real-time processing instead of traditional batch updates.
• Variety: the diversity of data formats, structured (tables), semi-structured (JSON, XML), and unstructured (text, images, videos, audio, logs).
• Veracity (sometimes added): the need to ensure trustworthiness and quality despite scale and diversity.

Before I came into the Big Data domain, in early 2010, my assumption was, standard SQL will not work for NoSQL databases, but I was 180 degrees wrong. NoSQL means Not-Only-SQL. Yeah, I felt dumb after knowing this lolz 😊.

In the past, we had only one Database Type i.e., RDBMS which here is referred to as SQL but with the introduction of Big Data, 4 more types of NoSQL databases came onto the surface i.e., Key-Value, Columnar, Document and Graph DB.

Examples of SQL database types are Oracle, DB2, Teradata, SQL Server, MySQL etc. which we all have been using for decades.

Hope we are on the same page that all SQL/RDBMS databases are just empty storage. The same is true with all these new NoSQL databases; these are again empty storages. In contrast with SQL/RDBMS databases which use server OS file systems like EXT, FAT21 etc. the new NoSQL database can additionally use HDFS.

Now, there is big confusion for storage selection in Hadoop as for a normal person, there are 5 new data stores in Hadoop i.e., 1) Hive 2) Key-Value DB (NoSQL) 3) Columnar DB (NoSQL) 4) Document DB (NoSQL) and 5) Graph DB (NoSQL). Yes, there are 5 data stores but there is 1 storage that is HDFS Hadoop Files Storage.

Data Swamp is a term famous as nightmare for CDO.

We all are aware of Data Lake or many other datastores where structure, semi and unstructured data is stored in an organized manner with proper Master Data, Metadata and Data Lineage. On the other hand, if datastores is missing Master Data, Metadata and Data Lineage, it becomes Data Swamp.

Data Swamp is a datastore that is not organized and from which it's very hard to extract information, which has a lot of duplications, and which has a lot of data quality issues.

In other words, retrieving and transforming data is no longer possible when it becomes a Data Swamp.

Data Swamps can happen quickly and can create a serious problem for companies relying on this storage, extraction, and transformation system for analytics.

Any organization which is in the process of becoming a Data-Driven Organization must first focus on Data Governance, Metadata Management, Master Data Management, Data Lineage. Having a data-driven organization with no clues like what and where data is residing is a nightmare and a waste of time and effort.

When we say Data is the most relevant demanding term in the current era then the question comes where to store it. There are 3 types of storage at the hardware level i.e., Object Storage, File Storage and Block Storage.

• Object Storage considers all types of data as an object. Here data is stored in a flat open storage area, with no folders or subfolders concept, along with its metadata and unique identifier which forms a storage pool keeping all details about that data near to it.
• File Storage keeps data as one piece of information inside folders and subfolders which is in a way is in Hierarchy mode, in contrast with Object storage e.g., you must remember how we used to keep manual papers in files in our school days. It’s the same concept here. So, when you need any specific data, you just need to know the location or path of the folder.
• Block storages consider a file as a single block of data and each block is stored as a separate piece of information. A unique address is given to each block for reference. Traditional databases are stored in blocks, and at the backend Hadoop files are also stored in blocks.

It refers to the staggering amount and variety of data produced by businesses every day. This is largely due to the variety of enterprise software, mobile apps, storage systems, and data formats each company relies on.

We have data in the shape of Video, Audio, Email, PDF, JSON, XML, Text etc. In other words, data is scattered in different formats and unless all formats are not converted in a standardized format, extracting information will also be challenging or time-consuming.

Imagine that you need to complete your taxes, but all your relevant papers are in different drawers, hidden in closets, and stuffed under couch cushions. Now imagine that you have multiple copies of the forms in these places, and some are written in Greek, while others are written in English and Spanish. How will you do your taxes, or clean your house for that matter, when this is the state of things? Unfortunately, this is a problem that is starting to plague companies across the world. This is Data Sprawl.

Data Assets can be anything that generates data for decision making which ultimately generates revenue for the organization.

We discussed Data Catalog, Data Dictionary and Data Glossary. Data Assets are the perfect Business Case for having the above-mentioned 3 Ds, as One must know what the data is all about and where it is residing. For Data-Driven companies, Data Assets are the most valuable artifacts which can be systems, websites, portals, mobile applications, output excel files, dashboards, reports, campaigns etc.

Facebook, Google, Twitter etc. are the best examples of which use its data as an Asset to run their businesses.

There are multiple ways to maintain Data Assets.

• Cutdown Cost of Data Management by deleting/ purging unnecessary data.
• Make max value out of it by selling it to Generate Revenue
• Manage Data Catalog
• Protection and Security Measures

‘In nutshell, Data Assets are those datasets which has ROI and are able to general revenue.’

Data Citizens is a very important role in Data field with respect to Data Science activities which still hasn’t been identified as a NORM, but time is not far when just like Data Analysts, Data Citizens will also become one of the MUST-TO-HAVE roles in every department.

‘Today, if 80%+- of Data Science projects are failing it's because of the absence of these Data Citizens roles.’

Data Analysts are functional consultants with skill sets like excel, databases using SQL, reports, and dashboards to dig and present data related to management KPI(s), later these analysis results are passed to ETL/ ELT developers or Data Engineers to implement data pipelines for Productionisation.

Data Citizens are also functional consultants with the machine learning skill sets to do initial data science analysis using DRAG & DROP tools to create, test and process machine learning models to run and predict KPI(s) for business and management, later these models are passed to Data Scientists to implement ML models for Productionisation.