Hadoop Architecture

Share this topic

Hadoop Architecture

From the book I Am Datapedia!, published here free by the author.

Part of I Am Datapedia!

Hadoop and its relationship with Big Data. The Big Data ecosystem has many components like NoSQL databases, In-Memory process, MPP, 3 Vs etc. but Hadoop is the backbone, and it was the main reason for the popularity of Big Data. In other words, Hadoop was the Selling Pitch for Salespeople.

'Hadoop is a Java-based framework.' Currently, its latest version is Apache Hadoop 3.3.4.’

Please note, Hadoop is not a solution, it's again an EMPTY storage. One must load data first just like we do in traditional databases before any analytics can be generated. For the first 5-7 years, Sales keep selling Big Data in such a way that you buy Big Data, and it will do magic for you, where all kinds of Analytics will be readily available but no it didn't happen, and you already bought it lolz. Learning: For every new technology, first hire a specialist internally and then invest 😊.

Let's go a little deeper.

All of you must have heard about Mainframe Systems which are big and expensive machines used for critical systems. Due to their high availability, fault tolerance, massive parallel process etc. only large organizations could afford those, plus those had huge datasets to play with.

Now, came the era of social media where data is generated in milliseconds. Only Twitter generates 52TB/day, as per status from somewhere, and there are mediums like Facebook, LinkedIn, Google, YouTube and many more. With this huge amount of data coming in, organizations needed capabilities like Mainframe but at a cheaper rate. There came Hadoop storage which could give you availability, fault tolerance, massive parallel process etc., on commodity hardware. Rather than relying on hardware to deliver high availability, the Hadoop library itself is designed to detect and handle failures at the application layer, so delivering a high-available service on top of a cluster of computers, each of which may be prone to failures.

Different terms in Hadoop architecture are HDFS, Name & Data Node, YARN, Resource & Node Manager, MapReduce, Replication Factor, High Availability and MPP (Massive Parallel Processing).

Express your reaction

Your Take on This Insight?

Share how this content resonates with you.

Comment
Useful0
Insightful0
Actionable0
Thought-provoking0
Inspiring0

0 reactionsSign in to react or comment

Comments (0)

No comments yet.

    Free account

    You can read every topic without one. A free account lets you react to this topic, comment under it and search the Knowledge Hub. Reacting and commenting need a confirmed e-mail address.