Once you sign in, your details will show here.
Knowledge Hub
Filter (2)
Sort
57 topics · Data Architecture · I Am Datapedia! · 10 shown
Show allEvery topic is free to read. A free account also lets you search the topics, react to a topic and comment under it.
- I Am Datapedia!
Data Security
Zero Trust Architecture
In today’s rapidly evolving digital landscape, security threats are more sophisticated and pervasive than ever. Traditional perimeter-based security is no longer sufficient to protect your organization’s most valuable asset – its data. This is where Zero Trust Architecture (ZTA) comes into play.
Data breaches can have catastrophic consequences, both financially and reputationally. Zero Trust Architecture ensures that no entity – human or machine – is trusted by default, minimizing the risk of unauthorized access. With millions of IoT devices and diverse infrastructures spanning on-premises and cloud environments, ZTA offers a robust security framework that addresses these complexities through stringent authentication, authorization, and entitlement mechanisms.
Starting with a zero-trust baseline where no one has access, the architecture gradually opens doors through verified credentials and continuous monitoring. This adaptive approach, combined with data encryption, masking, and anonymization, ensures that your defenses are resilient against emerging threats. Implementing Zero Trust Architecture significantly reduces the risk of data breaches, ensures regulatory compliance, streamlines security operations, and future-proofs your security strategy.
2 min readMore
- I Am Datapedia!
Data Architecture
Neon-inspired architecture
NEON-Inspired Architecture refers to a modern architectural approach that separates storage from compute, allowing systems to scale more efficiently and flexibly. The idea is inspired by cloud-native database designs where data storage and processing power are decoupled instead of tightly integrated.
In traditional database systems, storage and compute are bundled together. If you need more processing power, you often have to scale the entire system, including storage, even if storage capacity is sufficient. This creates inefficiencies in cost and scalability.
In a NEON-inspired approach, storage and compute operate independently. Storage is centralized and durable, while compute instances can scale up or down based on workload demand. This architecture supports elasticity, better resource utilization, and faster provisioning of environments.
One of the well-known implementations of this concept is Neon, which applies serverless principles to PostgreSQL by separating compute nodes from storage layers. While the term “NEON-inspired” may not refer exclusively to that company, it reflects the broader architectural pattern of disaggregation and cloud-native database design.
3 min readMore
- I Am Datapedia!
Data Architecture
On-Premises vs Cloud
On-Premises and Cloud both compete but at backend both operate the same way.
Since years and years Cloud is trying to make a mark but some or the other reason it couldn’t. Now things have changed, in recent era, Cloud has become arguably the biggest buzzword after the Covid-19 pandemic in 2019/20, as organizations had to enable Cloud services for their employees to support their businesses.
Let’s try to understand the core difference, to use IT solutions there are always two options either to have all the software(s) on On-Premises or on Cloud.
On-Premises: When an organization buys servers and infrastructure, and places those inside their own location or building under their own control, managed by their own internal teams, it is called On-Premises.
Cloud: Now when someone else places servers and infrastructure in their location or in their building and manages it for customer where customer accesses those servers and infrastructure via internet, it is called Cloud.
5 min readMore
- I Am Datapedia!
Data Architecture
IaaS(Infrastructure as A Service)
In IaaS, online compute power and storage is provided to customers in the shape of Virtual Machines (VS). Customers must bring their own software, tools and IT Solutions and then manage the environment, from installing/ upgrading software, HA, DR, compute power, storage standard etc., on its own. Cloud Service provider charges based on compute and storage usage.
Cloud example segregating IaaS:
If you only expect Me (Service Provider) to buy laptops (Infrastructure or Servers) with ONLY having Operating System of your choice and give you access and all rest of the activities like to install your special software(s) on it, install security software(s), enable software encryption, replaces normal hard disk with encryption disk, install all MS Office software(s) on it for daily activities, install database on it, configure Disaster Recovery and High Availability Strategy like taking backup on an external disk and keep moving data on periodic basic on second laptop so just in case if first laptop crashes, work can be continued on second laptop, periodic upgrading all the software(s) on both laptops etc. etc. etc. all will be take care by yourself is IaaS.
1 min readMore
- I Am Datapedia!
Data Architecture
CaaS(Container as A Service)
We have discussed differences between On-Premises and Cloud then IaaS where only Hardware is rented by Cloud providers. Next stage of this paradigm change is to rent Containers at a small scale, to save cost and for better management of resources. CaaS (Container as A Service) is one level lower than IaaS to rent resources on Cloud.
As the name says itself, CaaS (Container as A Service) is about deploying code in a container and deploying that container in Cloud. This saves us to manage hardware on our own in Cloud i.e., IaaS, so we can deploy our containers and underneath it, the hardware will be managed by Cloud providers. Most of the Cloud providers are offering
CaaS.
You are using Agile Methodology then you need CI/ CD (explained separately). In Agile, most of the time we have multiple teams working in silos on different modules on the same project and can be on the same deliverable. CaaS can be very helpful in this methodology where different teams are able to develop and deploy their code set any time using CI/ CD pipelines.
1 min readMore
- I Am Datapedia!
Data Architecture
PaaS(Platform as A Service)
As we move from IaaS to CaaS to PaaS, it becomes confusing. In IaaS and CaaS, we established that Infrastructure/ Servers and Storage is provided by Cloud providers and rest of all the activities are managed by customer team, itself. Now, let’s try to understand PaaS where all software(s) are made available for customers by the Cloud provider. Software(s) are already installed and ready to use. Customers don’t have to manage any compute or storage; everything is managed automatically by Cloud provider.
If you expect Me (Service Provider) to buy laptops (Infrastructure or Servers) with all the possible software(s) pre-installed and made those available to you via Internet, ready to be used in few clicks. Along with software(s) all rest of the activities like to install security software(s), enable software(s) encryption, replaces normal hard disk with encryption disk, install all MS Office software(s) on it for daily activities, install database on it, configure Disaster Recovery and High Availability Strategy like taking backup on an external disk and keep moving data on periodic basic on second laptop so just in case if first laptop crashes, work can be continued on second laptop, periodic upgrading all the software(s) on both laptops etc., all will be taken care by Me (Cloud Provider) is PaaS.
2 min readMore
- I Am Datapedia!
Data Architecture
FaaS (Function as A Service)
We have discussed IaaS and CaaS where only Infrastructure and Container level resources are rented on Cloud. Next stage of this paradigm change is to deploy Functions ONLY and pay for resources required to run those only. FaaS (Function as A Service) is one level lower than CaaS to rent resources on Cloud.
One of the biggest challenges to date is, whenever there is a change in the requirement, the whole code-set must redeploy. When code changes, it was always challenging to redeploy ONLY the changed code as there are always dependencies. Breaking the bigger code set into smaller pieces of code sets i.e., functions, can assist in mitigating this challenge wherever possible. Agree, in Data Pipelines it might be difficult but worth having this option to be utilized whenever possible.
As the name says itself, FaaS (Function as A Service) is about deploying ONLY Functions in Cloud. This saves us to manage Infrastructure on our own in Cloud, now we can deploy our Functions ONLY and underneath it the infrastructure will be managed by Cloud providers. Most of the Cloud providers are offering FaaS.
2 min readMore
- I Am Datapedia!
Data Architecture
SaaS(Software as A Service)
As we move from IaaS and PaaS to SaaS, it becomes very confusing segregating PaaS and SaaS, but please hold on to your horses’ 😊 we are going to nail it down in this topic.
To understand this example, pre-requisite topics are 1) ‘On-Promises vs Cloud’ and 2) IaaS and 3) PaaS.
In IaaS, we established that only Infrastructure/ Servers and Storage is provided by Cloud providers then in PaaS we establish along with Infrastructure/ Servers and Storage, Software(s) BUT EMPTY Software(s) are also made available along with rest of all the activities to be managed by the Cloud providers EXCEPT NO IMPLEMENTED SOLUTION.
In SaaS, there are pre-built or OOTB Solutions available in the Cloud. Please note, it’s not mandatory that all solutions are owned by Cloud providers, most of the solutions are created and maintained by the third parties. Third Parties use compute and storage from Cloud providers and build their own solutions and sell to customers. Now, customers just need to log-in into the Solution in Cloud as a Service that is why it’s called Software as A Service. A solution is owned and its management like installation/ upgradation, compute, storage, HA, DR etc., all are taken care of by the Solution Owner, again which can be by Cloud Providers or any Third Parties. Customers just need to subscribe to the SaaS service and start using the Solution.
2 min readMore
- I Am Datapedia!
Data Architecture
DaaS (Data as A Service)
Data As a Service (DaaS) is very near to SaaS service in Cloud. We know what SaaS is, it’s about pre-built or OOTB solutions available in the Cloud. Solution providers having their solutions in Cloud with API(s) enabled from where they allow customers to connect and pull data on demand who are then billed based on the data they pulled, this service is called DaaS.
For example, a company builds a solution on cloud where it pulls data from Facebook, Amazon, Ali Baba, etc., and using ML Modelling, predict customers buying behavior, customers surfing behavior, customers segmentation, products categorization, customers demographics etc., this analysis and analytics are then sold as DaaS to organizations in need for these details for marketing or sales planning etc.
The main difference between in SaaS and DaaS is, in SaaS, customer use whole vendor solution on Cloud e.g., customer logs in into vendor solution online and start using it whereas in DaaS service, customer can connect or subscribe to pull the data ONLY.
2 min readMore
- I Am Datapedia!
Data Architecture
AAaaS(Advance Analytics as A Service)
Please note, AAaaS has nothing to do with Cloud, it can be on On-Premises or On-Cloud. AAaaS is an environment setup for Data Scientists to play around with Machine Learning and Deep Learning Modelling.
Advance Analytics as a Service (AAaaS) is a new concept and still maturing, due to which it is still very confusing. Different companies are offering different approaches, so there is no one best approach.
Two approaches been followed.
- AAaaS as PaaS: Modelling tools like Databricks, Azure AML etc., are available as pre-installed services (refer to topic: PaaS) where customers can provision these tools in few clicks and simply start developing their Machine Learning or Deep Learning Models to build their own solutions. Here, customers must have their own MLOps along with CI/CD.
- AAaaS as SaaS: In this approach, Solution Providers hosts pre-built or OOTB Machine Learning and Deep Learning Models. These are offered to customers to consume only. Customers inject their own dataset for relevant data analysis and analytics. Here, customers must follow Solution Providers MLOps along with CI/CD.
1 min readMore
Build a strong foundation in Data & AI
Follow the Data Foundations learning path (Beginner to Advanced).