Knowledge Hub

Data Swamp is a term famous as nightmare for CDO.

We all are aware of Data Lake or many other datastores where structure, semi and unstructured data is stored in an organized manner with proper Master Data, Metadata and Data Lineage. On the other hand, if datastores is missing Master Data, Metadata and Data Lineage, it becomes Data Swamp.

Data Swamp is a datastore that is not organized and from which it's very hard to extract information, which has a lot of duplications, and which has a lot of data quality issues.

In other words, retrieving and transforming data is no longer possible when it becomes a Data Swamp.

Data Swamps can happen quickly and can create a serious problem for companies relying on this storage, extraction, and transformation system for analytics.

Any organization which is in the process of becoming a Data-Driven Organization must first focus on Data Governance, Metadata Management, Master Data Management, Data Lineage. Having a data-driven organization with no clues like what and where data is residing is a nightmare and a waste of time and effort.

HTAP stands for Hybrid Transactional/ Analytical Processing. It is a new architecture proposed by Garner in 2014.

In previous topics, we have established that there is an OLTP database for operational tasks and then Data Warehouse, Data Lake, Delta Lake, Data Lakehouse, Data/ Business Vault etc. for analysis and analysis. All keep data separately in their own storage for management decisions.

HTAP is proposing to use one system for both transaction and analytical workload. It is offering an in-memory plus columnar approach for analytical processes.

With the advancements in hardware, in-memory and Cloud, Gartner thinks it is very much possible to have both Transactional and Analytical processing on the same database.

Advantages

• No duplication of data as data will not move from the same one operational database.
• Business or Management can make decisions on actual data in real-time, in contrast if data must move from OLTP to OLAP systems.

Data Rooms or Datastore are both the same entity. Data Rooms are normally used for due diligence of confidential documents like handling Business Mergers or Acquisitions by bidders & advisers, compliance & audit teams.

Traditionally, there are Physical Data Rooms with Physical Documents. These Physical Data Rooms are for restricted people only, secured by guards with a maximum of one or two parties to go in at any given point-in-time which delays the process if multiple parties are using the same Physical Data Room.

With the growing businesses globally, it was a very expensive affair for multiple teams to fly over to be near these Physical Data Rooms. As an alternate and more viable solution, VDRs (Virtual Data Room) came into the picture which are available Online, behind Firewalls and with proper Access Control. VDRs have changed the game, now sellers can invite a bigger audience and teams sitting in one part of the world to participate in a bid process happening in the other part of the world. VDRs have cut down the cost and expedited the process.

As per the name depiction, Data Locality looks like knowing the location of the data; it is partially relevant.

Traditionally, to use data for analysis, at the backend the data is fetched from storage/ data nodes into master nodes, for computation. There is a complete history of how initially whole data was pulled then blocks were pulled then the columnar pull concept came-in, etc. so we won't go into detail about how technology has evolved to process data.

Since semi and unstructured data has become a necessary part of the Data World, it comes with huge storage requirements i.e., in TB and even in PB. To host this humongous data, clusters of 100s of storage/ data nodes are set up, so it is not possible to pull this size of data for computation in master nodes which are normally few. It was a critical requirement to process queries where data is residing.

With the introduction of Big Data/ Hadoop/ HDFS where data is stored in the shape of blocks, the concept surfaced i.e., instead of fetching data for computation, let's take the query to the storage/ data nodes, process it there and fetch only processed data. This process of moving the computation close to where the actual data resides on the storage/ data node, instead of moving large data to computation, is called Data Locality.

When we say Data is the most relevant demanding term in the current era then the question comes where to store it. There are 3 types of storage at the hardware level i.e., Object Storage, File Storage and Block Storage.

• Object Storage considers all types of data as an object. Here data is stored in a flat open storage area, with no folders or subfolders concept, along with its metadata and unique identifier which forms a storage pool keeping all details about that data near to it.
• File Storage keeps data as one piece of information inside folders and subfolders which is in a way is in Hierarchy mode, in contrast with Object storage e.g., you must remember how we used to keep manual papers in files in our school days. It’s the same concept here. So, when you need any specific data, you just need to know the location or path of the folder.
• Block storages consider a file as a single block of data and each block is stored as a separate piece of information. A unique address is given to each block for reference. Traditional databases are stored in blocks, and at the backend Hadoop files are also stored in blocks.

Hadoop and its relationship with Big Data. The Big Data ecosystem has many components like NoSQL databases, In-Memory process, MPP, 3 Vs etc. but Hadoop is the backbone, and it was the main reason for the popularity of Big Data. In other words, Hadoop was the Selling Pitch for Salespeople.

'Hadoop is a Java-based framework.' Currently, its latest version is Apache Hadoop 3.3.4.’

Please note, Hadoop is not a solution, it's again an EMPTY storage. One must load data first just like we do in traditional databases before any analytics can be generated. For the first 5-7 years, Sales keep selling Big Data in such a way that you buy Big Data, and it will do magic for you, where all kinds of Analytics will be readily available but no it didn't happen, and you already bought it lolz. Learning: For every new technology, first hire a specialist internally and then invest 😊.

Let's go a little deeper.

Obvious question, difference between Hadoop, HDFS and Hive?

Hadoop is an Open-Source Framework to store structured, semi/unstructured data. Default block size of 128MB (changeable).

HDFS is a Hadoop Files System just like FAT, NTFS, EXT etc.

Hive is only a wrapper on top of files stored in HDFS. Hive behaves just like tables in traditional databases, but it doesn't store any data, it just manages table structures on top of HDFS files. There are 2 ways Hive works, either store files in HDFS directly or create tables which will create HDFS files at the back end. The difference in these approaches is 1) each file stored in HDFS creates its own block so if there are 2 small files e.g., 50 MB each, then there will be 2 blocks created. 2) on the other hand, Hive manages its own blocks so if there are 2 small files of e.g., 50 MB each, then Hive will store both in a single block.

Consider 1000s of 1MB files float in from social media means 1000s of 1MB small blocks created impacting MPP performance. The best practice is to have blocks of 128MB.

It refers to the staggering amount and variety of data produced by businesses every day. This is largely due to the variety of enterprise software, mobile apps, storage systems, and data formats each company relies on.

We have data in the shape of Video, Audio, Email, PDF, JSON, XML, Text etc. In other words, data is scattered in different formats and unless all formats are not converted in a standardized format, extracting information will also be challenging or time-consuming.

Imagine that you need to complete your taxes, but all your relevant papers are in different drawers, hidden in closets, and stuffed under couch cushions. Now imagine that you have multiple copies of the forms in these places, and some are written in Greek, while others are written in English and Spanish. How will you do your taxes, or clean your house for that matter, when this is the state of things? Unfortunately, this is a problem that is starting to plague companies across the world. This is Data Sprawl.

Dormant Data and Dark Data are mostly confused with each other.

Dark Data

‘Data that is not utilized is Dark Data.’

Dark Data is data that is not in use due to lack of technology and is not a Data Asset. Dormant Data is the data, which is useable, but decision-makers don't use it, or it’s not required for analysis but that doesn't mean you can delete it or purge it, so we keep it in Tape drives, in Hadoop and nowadays Object Storage is also good choice.

Before the introduction of Big Data, the % age of Dark Data was around 90% as it included semi and unstructured datasets and sometimes structured as well. For example, JSON or XML files, Video or Audio files, Logs, Metadata etc., all these datasets were there but technologies were either not there or the affair to analyze those were very expensive. The only reason companies kept this kind of data was for Compliance or Audit purposes. Since the Big Data ecosystem has come onto the surface and hardware has become dead cheap so hosting and processing these datasets has become possible.

As we discussed in topic i.e., Dark Data, that it is utmost important for a CDO (Chief Data Officer) to minimize Dark Data as much as possible, in the organization. In other words, the more the Dark Data, the more the cost without any ROI. Time is not far when Dark Data Management will become the most essential and obvious KPI for CDO (Chief Data Officer).

Now the questions are.

• Is all Dark Data bad? NO.
• Is all Dark Data delete able or purge able? Yes and No.

‘Data Detritus means that data, which is no more required, which is of no use anymore etc.’

• Dark data is something which was and is, not in-use.
• Dormant data is something which is hardly in-use.
• Detritus data is something which was in-use but will not be in-use, anymore.

In other words, Detritus data was once valuable before it became Dormant, as compared to traditional Dark Data (explained separately) which was never in-use due to technologies barriers etc. Detritus data is normally not deleted or purged as it can still be required for Regulatory or Compliance parties.