Kappa Architecture

Kappa Architecture is mostly confused with Lambda Architecture which is fair as both processes Streaming as well as Batch data. Then how both are different?

Due to high demand of Real-Time data, Kappa Architecture has emerged but not for all kinds of data sizes just like Lambda Architecture is.

We have gone through in detail (in separate topic) like what is Lambda Architecture so won’t go in details like how Streaming and Batch works. Let’s understand how Kappa is different than Lambda Architecture.

Kappa Architecture offers ONLY stream processing using Queuing tools like KAFKA for both Streaming and Batch datasets. It doesn’t offer separate tools or technologies to process batches. It says, process both Real-Time Streaming and Batch data using same tool as KAFKA but the biggest differentiator here is, it stores data in a separate data store which might not be suitable to store TBs or PBs of dataset. It offers batch related downstream applications to connect directly to its storage.

So, if your organization has TBs or potentially PBs of data then go for Lambda Architecture and store data in Hadoop or Object Storage for Analysis and Analytics but if you have small or medium size data with no historical data maintenance requirement then can use Kappa Architecture.