Knowledge Hub

Data Swamp is a term famous as nightmare for CDO.

We all are aware of Data Lake or many other datastores where structure, semi and unstructured data is stored in an organized manner with proper Master Data, Metadata and Data Lineage. On the other hand, if datastores is missing Master Data, Metadata and Data Lineage, it becomes Data Swamp.

Data Swamp is a datastore that is not organized and from which it's very hard to extract information, which has a lot of duplications, and which has a lot of data quality issues.

In other words, retrieving and transforming data is no longer possible when it becomes a Data Swamp.

Data Swamps can happen quickly and can create a serious problem for companies relying on this storage, extraction, and transformation system for analytics.

Any organization which is in the process of becoming a Data-Driven Organization must first focus on Data Governance, Metadata Management, Master Data Management, Data Lineage. Having a data-driven organization with no clues like what and where data is residing is a nightmare and a waste of time and effort.

HTAP stands for Hybrid Transactional/ Analytical Processing. It is a new architecture proposed by Garner in 2014.

In previous topics, we have established that there is an OLTP database for operational tasks and then Data Warehouse, Data Lake, Delta Lake, Data Lakehouse, Data/ Business Vault etc. for analysis and analysis. All keep data separately in their own storage for management decisions.

HTAP is proposing to use one system for both transaction and analytical workload. It is offering an in-memory plus columnar approach for analytical processes.

With the advancements in hardware, in-memory and Cloud, Gartner thinks it is very much possible to have both Transactional and Analytical processing on the same database.

Advantages

• No duplication of data as data will not move from the same one operational database.
• Business or Management can make decisions on actual data in real-time, in contrast if data must move from OLTP to OLAP systems.

As per the name depiction, Data Locality looks like knowing the location of the data; it is partially relevant.

Traditionally, to use data for analysis, at the backend the data is fetched from storage/ data nodes into master nodes, for computation. There is a complete history of how initially whole data was pulled then blocks were pulled then the columnar pull concept came-in, etc. so we won't go into detail about how technology has evolved to process data.

Since semi and unstructured data has become a necessary part of the Data World, it comes with huge storage requirements i.e., in TB and even in PB. To host this humongous data, clusters of 100s of storage/ data nodes are set up, so it is not possible to pull this size of data for computation in master nodes which are normally few. It was a critical requirement to process queries where data is residing.

With the introduction of Big Data/ Hadoop/ HDFS where data is stored in the shape of blocks, the concept surfaced i.e., instead of fetching data for computation, let's take the query to the storage/ data nodes, process it there and fetch only processed data. This process of moving the computation close to where the actual data resides on the storage/ data node, instead of moving large data to computation, is called Data Locality.

Hadoop and its relationship with Big Data. The Big Data ecosystem has many components like NoSQL databases, In-Memory process, MPP, 3 Vs etc. but Hadoop is the backbone, and it was the main reason for the popularity of Big Data. In other words, Hadoop was the Selling Pitch for Salespeople.

'Hadoop is a Java-based framework.' Currently, its latest version is Apache Hadoop 3.3.4.’

Please note, Hadoop is not a solution, it's again an EMPTY storage. One must load data first just like we do in traditional databases before any analytics can be generated. For the first 5-7 years, Sales keep selling Big Data in such a way that you buy Big Data, and it will do magic for you, where all kinds of Analytics will be readily available but no it didn't happen, and you already bought it lolz. Learning: For every new technology, first hire a specialist internally and then invest 😊.

Let's go a little deeper.

Obvious question, difference between Hadoop, HDFS and Hive?

Hadoop is an Open-Source Framework to store structured, semi/unstructured data. Default block size of 128MB (changeable).

HDFS is a Hadoop Files System just like FAT, NTFS, EXT etc.

Hive is only a wrapper on top of files stored in HDFS. Hive behaves just like tables in traditional databases, but it doesn't store any data, it just manages table structures on top of HDFS files. There are 2 ways Hive works, either store files in HDFS directly or create tables which will create HDFS files at the back end. The difference in these approaches is 1) each file stored in HDFS creates its own block so if there are 2 small files e.g., 50 MB each, then there will be 2 blocks created. 2) on the other hand, Hive manages its own blocks so if there are 2 small files of e.g., 50 MB each, then Hive will store both in a single block.

Consider 1000s of 1MB files float in from social media means 1000s of 1MB small blocks created impacting MPP performance. The best practice is to have blocks of 128MB.

Data Island has kind of disappeared as integration technologies have matured.

‘Data Island is a datastore(s) which is/ are not accessible.’

In previous eras when network security was a big concern or even before, when network connectivity was not even there, the datastores were considered as Data Island as external integration was not possible.

Let’s try to understand with examples. We all know WIFI has become a norm in our lives and living without it is like living without Air right? In past when we used to visit a friend’s place we used to greet first, now we first ask, ‘What’s your WIFI Password’ lolz 😊?’. But if you are as old as I am, before 2005 WIFI was not a thing. There were ONLY cables which used to connect our desktops or laptops with network, well I still prefer cable. During that era consider a datastore sitting in different city or region then how to connect? In that situation, we had floppy drives for small data transfer (yeah, I won’t explain what is floppy drive lolz 😊, old folks know what I am talking about. Let’s let newbies find out what is floppy drive), and for huge data we used to have big hard disks and sometimes desktops itself were moved to move data from one location to another location. This datastore, which was not connectable with other datastore, was call Data Island.

Data Silos is always confused with Data Island.

Data Islands are those datastores which are not accessible from external world may be due technology barriers, but Data Silos are those which are only accessible for their relevant stakeholders.

There can be a lot of reasons why data is in silo.

• A particular datastore might be hosting confidential data e.g., finance data, audit data etc.
• A particular data which is not readable e.g., semi or unstructured data sets.
• There is Security or Privacy concerns, so data is not shared among another departments.
• The quality of data is so low that it can’t be shared with management as it may cause wrong decisions and may damage the company’s image.
• Sometime, in development environment different database instances are created for different development teams so if one changes data or table’s structures, other development teams are not impacted.

Why should Data Silos not exist?

There are four types of instructions to pull data i.e., SISD, SIMD, MISD and MIMD.

• SISD: Single Instruction Stream and Single Data Stream
• SIMD: Single Instruction Stream and Multiple Data Stream
• MISD: Multiple Instruction Stream and Single Data Stream
• MIMD: Multiple Instruction Stream and Multiple Data Stream

Before we jump into understanding all above mentioned techniques, let’s first understand what is a Core in a machine or in a computer or in a server etc.?

A CPU Core is built of billions of microscopic transistors to process instructions passed to it. One instruction is handled by One core, if someone needs multitasking or multiple instruction in parallel, minimum 2 cores are required.

We know when we insert or load data in a computer or server or in a node, it gets saved in hard disk. We won’t go in to details how it is stored on the hard disk. We can imagine when we write something on paper, consider the same way, data is saved in a hard disk.

Data Serving is also known as the Serving Layer and it has been there since its inception but as technologies have matured, many new channels have been introduced to support this layer.

‘Serving Layer is last component of the Lambda Architecture.’

Lambda Architecture has been explained in separate topic, so we won’t go into details of it but referring to it, as mentioned above, it is the last layer which is exposed to End Users for reporting purposes. This Layer is, if not all the time but majorly designed based on business requirements.

We all know OLTP systems are not meant to be exposed for reporting purposes. In the current era, for reporting, there are many data stores available i.e., Data Warehouse, OLAP, Data Lake, NoSQL, Semantic layers but ideally Semantic Layer should be treated as Serving Layer.

To be more specific

• If Serving Layer is designed for specific business requirement then Semantic Layer is last layer.
• If the Serving Layer is designed for generic use then it can be Data Warehouse, OLAP, Data Lake, NoSQL etc.

Consumption layer is confused with Serving Layer due to its name. Folks relates it that the layer which is used by Consumers to extract data. The Layer which Consumers use to extract information is Serving layer (explained separately).

Consumption Layer, as its name says it all, is a layer which consumes or takes-in data from source systems. In the current era, due to huge amounts of data coming in every second from sources like social media, CCTV, images, audios, sensors data etc. technology was required to consume data on a fast pace in MPP mode.

Tools like Hadoop, Object Storage and many RDBMS are giving MPP features now a days.

In Lambda Architecture, Serving Layer is the last layer, Consumption Layer is the layer which process data from source till Serving layer.

In the image, Batch and Speed Layers are Consumption Layers. In an OLAP system, the consumption layer is Data Warehouse and Data Mart, considering there are OLAP Cubes and Semantic layers but in a Data Warehouse, consumption layer is Landing and Staging zone.