Knowledge Hub

Before I came into the Big Data domain, in early 2010, my assumption was, standard SQL will not work for NoSQL databases, but I was 180 degrees wrong. NoSQL means Not-Only-SQL. Yeah, I felt dumb after knowing this lolz 😊.

In the past, we had only one Database Type i.e., RDBMS which here is referred to as SQL but with the introduction of Big Data, 4 more types of NoSQL databases came onto the surface i.e., Key-Value, Columnar, Document and Graph DB.

Examples of SQL database types are Oracle, DB2, Teradata, SQL Server, MySQL etc. which we all have been using for decades.

Hope we are on the same page that all SQL/RDBMS databases are just empty storage. The same is true with all these new NoSQL databases; these are again empty storages. In contrast with SQL/RDBMS databases which use server OS file systems like EXT, FAT21 etc. the new NoSQL database can additionally use HDFS.

Now, there is big confusion for storage selection in Hadoop as for a normal person, there are 5 new data stores in Hadoop i.e., 1) Hive 2) Key-Value DB (NoSQL) 3) Columnar DB (NoSQL) 4) Document DB (NoSQL) and 5) Graph DB (NoSQL). Yes, there are 5 data stores but there is 1 storage that is HDFS Hadoop Files Storage.

Data Mesh is normally confused with Data Mashup (explained separately), but both are totally different.

Data Mesh is a modern approach to data architecture and governance that emphasizes decentralization and domain-driven ownership of data products. Unlike traditional centralized data systems, which rely on a monolithic team to manage and deliver data assets across an organization, Data Mesh shifts the focus towards empowering individual domains to become autonomous in managing their data products. This paradigm fosters scalability, improves data usability, and ensures alignment with specific business needs.

Understanding the Foundations of Data Mesh

At its core, Data Mesh revolves around four key principles:

1. Domain-Oriented Decentralization: The fundamental premise of Data Mesh is the distribution of data ownership to individual domains. A domain, in this context, refers to a specific business area or operational unit, such as marketing, sales, or customer service. Each domain is responsible for defining, managing, and maintaining its own data products. By doing so, Data Mesh reduces the bottlenecks and inefficiencies associated with centralized data teams.
2. Data as a Product (DAaP): Data Mesh introduces the concept of treating data as a product. This means that data products must be designed with the same rigor and customer-centricity as any software product. Each data product must:

Data Swamp is a term famous as nightmare for CDO.

We all are aware of Data Lake or many other datastores where structure, semi and unstructured data is stored in an organized manner with proper Master Data, Metadata and Data Lineage. On the other hand, if datastores is missing Master Data, Metadata and Data Lineage, it becomes Data Swamp.

Data Swamp is a datastore that is not organized and from which it's very hard to extract information, which has a lot of duplications, and which has a lot of data quality issues.

In other words, retrieving and transforming data is no longer possible when it becomes a Data Swamp.

Data Swamps can happen quickly and can create a serious problem for companies relying on this storage, extraction, and transformation system for analytics.

Any organization which is in the process of becoming a Data-Driven Organization must first focus on Data Governance, Metadata Management, Master Data Management, Data Lineage. Having a data-driven organization with no clues like what and where data is residing is a nightmare and a waste of time and effort.

HTAP stands for Hybrid Transactional/ Analytical Processing. It is a new architecture proposed by Garner in 2014.

In previous topics, we have established that there is an OLTP database for operational tasks and then Data Warehouse, Data Lake, Delta Lake, Data Lakehouse, Data/ Business Vault etc. for analysis and analysis. All keep data separately in their own storage for management decisions.

HTAP is proposing to use one system for both transaction and analytical workload. It is offering an in-memory plus columnar approach for analytical processes.

With the advancements in hardware, in-memory and Cloud, Gartner thinks it is very much possible to have both Transactional and Analytical processing on the same database.

Advantages

• No duplication of data as data will not move from the same one operational database.
• Business or Management can make decisions on actual data in real-time, in contrast if data must move from OLTP to OLAP systems.

Data Rooms or Datastore are both the same entity. Data Rooms are normally used for due diligence of confidential documents like handling Business Mergers or Acquisitions by bidders & advisers, compliance & audit teams.

Traditionally, there are Physical Data Rooms with Physical Documents. These Physical Data Rooms are for restricted people only, secured by guards with a maximum of one or two parties to go in at any given point-in-time which delays the process if multiple parties are using the same Physical Data Room.

With the growing businesses globally, it was a very expensive affair for multiple teams to fly over to be near these Physical Data Rooms. As an alternate and more viable solution, VDRs (Virtual Data Room) came into the picture which are available Online, behind Firewalls and with proper Access Control. VDRs have changed the game, now sellers can invite a bigger audience and teams sitting in one part of the world to participate in a bid process happening in the other part of the world. VDRs have cut down the cost and expedited the process.

As per the name depiction, Data Locality looks like knowing the location of the data; it is partially relevant.

Traditionally, to use data for analysis, at the backend the data is fetched from storage/ data nodes into master nodes, for computation. There is a complete history of how initially whole data was pulled then blocks were pulled then the columnar pull concept came-in, etc. so we won't go into detail about how technology has evolved to process data.

Since semi and unstructured data has become a necessary part of the Data World, it comes with huge storage requirements i.e., in TB and even in PB. To host this humongous data, clusters of 100s of storage/ data nodes are set up, so it is not possible to pull this size of data for computation in master nodes which are normally few. It was a critical requirement to process queries where data is residing.

With the introduction of Big Data/ Hadoop/ HDFS where data is stored in the shape of blocks, the concept surfaced i.e., instead of fetching data for computation, let's take the query to the storage/ data nodes, process it there and fetch only processed data. This process of moving the computation close to where the actual data resides on the storage/ data node, instead of moving large data to computation, is called Data Locality.

When we say Data is the most relevant demanding term in the current era then the question comes where to store it. There are 3 types of storage at the hardware level i.e., Object Storage, File Storage and Block Storage.

• Object Storage considers all types of data as an object. Here data is stored in a flat open storage area, with no folders or subfolders concept, along with its metadata and unique identifier which forms a storage pool keeping all details about that data near to it.
• File Storage keeps data as one piece of information inside folders and subfolders which is in a way is in Hierarchy mode, in contrast with Object storage e.g., you must remember how we used to keep manual papers in files in our school days. It’s the same concept here. So, when you need any specific data, you just need to know the location or path of the folder.
• Block storages consider a file as a single block of data and each block is stored as a separate piece of information. A unique address is given to each block for reference. Traditional databases are stored in blocks, and at the backend Hadoop files are also stored in blocks.

Hadoop and its relationship with Big Data. The Big Data ecosystem has many components like NoSQL databases, In-Memory process, MPP, 3 Vs etc. but Hadoop is the backbone, and it was the main reason for the popularity of Big Data. In other words, Hadoop was the Selling Pitch for Salespeople.

'Hadoop is a Java-based framework.' Currently, its latest version is Apache Hadoop 3.3.4.’

Please note, Hadoop is not a solution, it's again an EMPTY storage. One must load data first just like we do in traditional databases before any analytics can be generated. For the first 5-7 years, Sales keep selling Big Data in such a way that you buy Big Data, and it will do magic for you, where all kinds of Analytics will be readily available but no it didn't happen, and you already bought it lolz. Learning: For every new technology, first hire a specialist internally and then invest 😊.

Let's go a little deeper.

Obvious question, difference between Hadoop, HDFS and Hive?

Hadoop is an Open-Source Framework to store structured, semi/unstructured data. Default block size of 128MB (changeable).

HDFS is a Hadoop Files System just like FAT, NTFS, EXT etc.

Hive is only a wrapper on top of files stored in HDFS. Hive behaves just like tables in traditional databases, but it doesn't store any data, it just manages table structures on top of HDFS files. There are 2 ways Hive works, either store files in HDFS directly or create tables which will create HDFS files at the back end. The difference in these approaches is 1) each file stored in HDFS creates its own block so if there are 2 small files e.g., 50 MB each, then there will be 2 blocks created. 2) on the other hand, Hive manages its own blocks so if there are 2 small files of e.g., 50 MB each, then Hive will store both in a single block.

Consider 1000s of 1MB files float in from social media means 1000s of 1MB small blocks created impacting MPP performance. The best practice is to have blocks of 128MB.

It refers to the staggering amount and variety of data produced by businesses every day. This is largely due to the variety of enterprise software, mobile apps, storage systems, and data formats each company relies on.

We have data in the shape of Video, Audio, Email, PDF, JSON, XML, Text etc. In other words, data is scattered in different formats and unless all formats are not converted in a standardized format, extracting information will also be challenging or time-consuming.

Imagine that you need to complete your taxes, but all your relevant papers are in different drawers, hidden in closets, and stuffed under couch cushions. Now imagine that you have multiple copies of the forms in these places, and some are written in Greek, while others are written in English and Spanish. How will you do your taxes, or clean your house for that matter, when this is the state of things? Unfortunately, this is a problem that is starting to plague companies across the world. This is Data Sprawl.