Knowledge Hub

On-Premises and Cloud both compete but at backend both operate the same way.

Since years and years Cloud is trying to make a mark but some or the other reason it couldn’t. Now things have changed, in recent era, Cloud has become arguably the biggest buzzword after the Covid-19 pandemic in 2019/20, as organizations had to enable Cloud services for their employees to support their businesses.

Let’s try to understand the core difference, to use IT solutions there are always two options either to have all the software(s) on On-Premises or on Cloud.

On-Premises: When an organization buys servers and infrastructure, and places those inside their own location or building under their own control, managed by their own internal teams, it is called On-Premises.

Cloud: Now when someone else places servers and infrastructure in their location or in their building and manages it for customer where customer accesses those servers and infrastructure via internet, it is called Cloud.

In IaaS, online compute power and storage is provided to customers in the shape of Virtual Machines (VS). Customers must bring their own software, tools and IT Solutions and then manage the environment, from installing/ upgrading software, HA, DR, compute power, storage standard etc., on its own. Cloud Service provider charges based on compute and storage usage.

Cloud example segregating IaaS:

If you only expect Me (Service Provider) to buy laptops (Infrastructure or Servers) with ONLY having Operating System of your choice and give you access and all rest of the activities like to install your special software(s) on it, install security software(s), enable software encryption, replaces normal hard disk with encryption disk, install all MS Office software(s) on it for daily activities, install database on it, configure Disaster Recovery and High Availability Strategy like taking backup on an external disk and keep moving data on periodic basic on second laptop so just in case if first laptop crashes, work can be continued on second laptop, periodic upgrading all the software(s) on both laptops etc. etc. etc. all will be take care by yourself is IaaS.

We have discussed differences between On-Premises and Cloud then IaaS where only Hardware is rented by Cloud providers. Next stage of this paradigm change is to rent Containers at a small scale, to save cost and for better management of resources. CaaS (Container as A Service) is one level lower than IaaS to rent resources on Cloud.

As the name says itself, CaaS (Container as A Service) is about deploying code in a container and deploying that container in Cloud. This saves us to manage hardware on our own in Cloud i.e., IaaS, so we can deploy our containers and underneath it, the hardware will be managed by Cloud providers. Most of the Cloud providers are offering CaaS.

You are using Agile Methodology then you need CI/ CD (explained separately). In Agile, most of the time we have multiple teams working in silos on different modules on the same project and can be on the same deliverable. CaaS can be very helpful in this methodology where different teams are able to develop and deploy their code set any time using CI/ CD pipelines.

As we move from IaaS to CaaS to PaaS, it becomes confusing. In IaaS and CaaS, we established that Infrastructure/ Servers and Storage is provided by Cloud providers and rest of all the activities are managed by customer team, itself. Now, let’s try to understand PaaS where all software(s) are made available for customers by the Cloud provider. Software(s) are already installed and ready to use. Customers don’t have to manage any compute or storage; everything is managed automatically by Cloud provider.

If you expect Me (Service Provider) to buy laptops (Infrastructure or Servers) with all the possible software(s) pre-installed and made those available to you via Internet, ready to be used in few clicks. Along with software(s) all rest of the activities like to install security software(s), enable software(s) encryption, replaces normal hard disk with encryption disk, install all MS Office software(s) on it for daily activities, install database on it, configure Disaster Recovery and High Availability Strategy like taking backup on an external disk and keep moving data on periodic basic on second laptop so just in case if first laptop crashes, work can be continued on second laptop, periodic upgrading all the software(s) on both laptops etc., all will be taken care by Me (Cloud Provider) is PaaS.

We have discussed IaaS and CaaS where only Infrastructure and Container level resources are rented on Cloud. Next stage of this paradigm change is to deploy Functions ONLY and pay for resources required to run those only. FaaS (Function as A Service) is one level lower than CaaS to rent resources on Cloud.

One of the biggest challenges to date is, whenever there is a change in the requirement, the whole code-set must redeploy. When code changes, it was always challenging to redeploy ONLY the changed code as there are always dependencies. Breaking the bigger code set into smaller pieces of code sets i.e., functions, can assist in mitigating this challenge wherever possible. Agree, in Data Pipelines it might be difficult but worth having this option to be utilized whenever possible.

As the name says itself, FaaS (Function as A Service) is about deploying ONLY Functions in Cloud. This saves us to manage Infrastructure on our own in Cloud, now we can deploy our Functions ONLY and underneath it the infrastructure will be managed by Cloud providers. Most of the Cloud providers are offering FaaS.

As we move from IaaS and PaaS to SaaS, it becomes very confusing segregating PaaS and SaaS, but please hold on to your horses’ 😊 we are going to nail it down in this topic.

To understand this example, pre-requisite topics are 1) ‘On-Promises vs Cloud’ and 2) IaaS and 3) PaaS.

In IaaS, we established that only Infrastructure/ Servers and Storage is provided by Cloud providers then in PaaS we establish along with Infrastructure/ Servers and Storage, Software(s) BUT EMPTY Software(s) are also made available along with rest of all the activities to be managed by the Cloud providers EXCEPT NO IMPLEMENTED SOLUTION.

In SaaS, there are pre-built or OOTB Solutions available in the Cloud. Please note, it’s not mandatory that all solutions are owned by Cloud providers, most of the solutions are created and maintained by the third parties. Third Parties use compute and storage from Cloud providers and build their own solutions and sell to customers. Now, customers just need to log-in into the Solution in Cloud as a Service that is why it’s called Software as A Service. A solution is owned and its management like installation/ upgradation, compute, storage, HA, DR etc., all are taken care of by the Solution Owner, again which can be by Cloud Providers or any Third Parties. Customers just need to subscribe to the SaaS service and start using the Solution.

In the current era there are many data stores like Database, Data Warehouse, Data Marts, Big Data, Data Lake, Data Vault, Business Vault, SQL, and NoSQL etc. Use of all these terms is different with their own horizon, their own skill sets whereas tools and languages can be the same for all of these.

First one must understand is that Database is a datastore, we can’t link it to any data modelling solution. In other words, OLTP or Data Warehouse or Data Mart or OLAP all use databases as a medium to store their datasets. Additionally, Databases can be SQL or NoSQL.

Now, OLTP is on one side of the line and Data Warehouse/ Data Marts and Big Data/ Data Lake etc. are on the other side. Normally, databases are called OLTP (Online Transactional Processing) whereas Data Warehouse/ Data Marts fall under the term OLAP (Online Analytical Processing).

As far as Big Data/ Data Lake is concerned, Data World has yet to tag these with either OLTP or OLAP but for sure these are not meant for OLTP. At the same time NoSQL is stepping into OLTP zone and is supporting OLTP as well.

The term Big Data emerged in the early 2000s as enterprises began facing unprecedented growth in the amount, speed, and diversity of information being generated. Traditional databases and warehouses, while reliable for structured data, were not designed to handle this new reality. Big Data describes not only the massive scale of data itself but also the ecosystem of technologies, methods, and practices developed to capture, store, and analyze it.

At the heart of Big Data is the concept of the “three Vs”, later extended to four or more:

• Volume: the enormous scale of data generated from transactions, machines, sensors, mobile devices, and social platforms.
• Velocity: the speed at which data arrives, often requiring real-time or near real-time processing instead of traditional batch updates.
• Variety: the diversity of data formats, structured (tables), semi-structured (JSON, XML), and unstructured (text, images, videos, audio, logs).
• Veracity (sometimes added): the need to ensure trustworthiness and quality despite scale and diversity.

Before I came into the Big Data domain, in early 2010, my assumption was, standard SQL will not work for NoSQL databases, but I was 180 degrees wrong. NoSQL means Not-Only-SQL. Yeah, I felt dumb after knowing this lolz 😊.

In the past, we had only one Database Type i.e., RDBMS which here is referred to as SQL but with the introduction of Big Data, 4 more types of NoSQL databases came onto the surface i.e., Key-Value, Columnar, Document and Graph DB.

Examples of SQL database types are Oracle, DB2, Teradata, SQL Server, MySQL etc. which we all have been using for decades.

Hope we are on the same page that all SQL/RDBMS databases are just empty storage. The same is true with all these new NoSQL databases; these are again empty storages. In contrast with SQL/RDBMS databases which use server OS file systems like EXT, FAT21 etc. the new NoSQL database can additionally use HDFS.

Now, there is big confusion for storage selection in Hadoop as for a normal person, there are 5 new data stores in Hadoop i.e., 1) Hive 2) Key-Value DB (NoSQL) 3) Columnar DB (NoSQL) 4) Document DB (NoSQL) and 5) Graph DB (NoSQL). Yes, there are 5 data stores but there is 1 storage that is HDFS Hadoop Files Storage.

When we say Data is the most relevant demanding term in the current era then the question comes where to store it. There are 3 types of storage at the hardware level i.e., Object Storage, File Storage and Block Storage.

• Object Storage considers all types of data as an object. Here data is stored in a flat open storage area, with no folders or subfolders concept, along with its metadata and unique identifier which forms a storage pool keeping all details about that data near to it.
• File Storage keeps data as one piece of information inside folders and subfolders which is in a way is in Hierarchy mode, in contrast with Object storage e.g., you must remember how we used to keep manual papers in files in our school days. It’s the same concept here. So, when you need any specific data, you just need to know the location or path of the folder.
• Block storages consider a file as a single block of data and each block is stored as a separate piece of information. A unique address is given to each block for reference. Traditional databases are stored in blocks, and at the backend Hadoop files are also stored in blocks.