Knowledge Hub

To start with, ACID and BaSE both are Data Consistency models which compete on a use case requirement basis. In the previous topic, we discussed ACID, let’s discuss BaSE. What is BaSE? Never heard about it right 😊.

The basic difference between ACID and BaSE models is the way they deal with this limitation.

• The ACID model provides a consistent system.
• The BASE model provides high availability.

‘ACID is about Atomicity, Consistency, Isolation and Durability.’

‘BaSE is about Basically Availability, Soft State and Eventually Consistent.’

There are 3 traits of BaSE i.e., basically available, soft state and eventually consistent.

• Basically Available: The system is guaranteed to be available in the event of failure.
• Soft State: Due to the lack of immediate consistency, data values may change over time. The BaSE model breaks off with the concept of a database that enforces its consistency, delegating that responsibility to developers.
• Eventually Consistent: The fact that BASE does not enforce immediate consistency does not mean that it never achieves it. However, until it does, data reads are still possible (even though they might not reflect reality).

Starting with DevOps, we all know Waterfall and Agile methodologies. We won’t go into details of these methodologies but for audience comfort, DevOps is mostly about operationalizing Agile processes where a set of practices combines software development (Dev) and IT Operations (Ops) to make i.e., DevOps.

‘DevOps is a physical face Agile.’

‘DevOps includes Plan, Create, Verify, Package, Release, Configure, Monitor, and Version Control.’

In DevOps, activities are mainly executed based on use cases, user stories and tasks, there are also other practices conducted by different organizations.

Engineers in DevOps independently accomplish tasks (for example, deploying code or provisioning infrastructure) that normally would have required assistance from other teams. Responsibilities are shared by all team members.

Please note, when we say Engineers in DevOps independently accomplish all tasks, it can only happen if most of the activities are automated from provisioning of resources, services, VMs, compute, memory, testing, integration testing, training, deployment etc.

You must be confused between DevOps and CI/ CD.

‘CI stands for Continuous Integration and CD stands for Continuous Delivery/ Deployment.’

We know, in development and implement life cycle there are multiple steps involved e.g., development of code, packaging it, testing it, and then deploying in production.

The challenge in traditional approach is, once development is completed rest of the steps in above cycle were very time consuming as there are a lot of manual processes involve plus dependencies on many departments for review, approval, and deployment manually. Due to manual processes, there was always risk of human errors. Many a times, we have seen one deployment can affect other’s code as well. For example, there are 5 deployments conducted at the same time, 4 deployments went smooth but 5th one impact last 4 as well. We won’t go into details but trust me, I have personally seen the above incidents many times. OK OK OK, yes once me deployment screwed other deployments as well lolz 😊. The point is whenever there is human involvement, errors are inevitable.

DevSecOps is nothing less DevOps but with addition step of security.

Due to the quick and fast deployment cycles, security has become an alarming factor in DevOps, so security has been included right before the creation process and therefore DevSecOps (development, security, and operations) came onto the surface.

DataOps follows the same principles just like DevOps/ DevSecOps but here it has few more elements related to Data domain.

DataOps focuses on data related aspects of operations. It makes sure data is properly integrated with source and target, its flowing-in using automated and best practices data pipelines, its highly secured, it’s of high quality etc.

DevOps doesn’t care whether its data project or mobile application project or any other technology project, but DataOps focuses on Data like where it is coming and where it will go, why it is coming and how it will be used, how much it costed and how much it will monetize etc.

As mentioned above, DataOps includes all the steps or processes involved in DevOps/ DevSecOps but at the same time it also introduces a few additional steps.

DataOps Process Flow

• Regular Integration
• Regular Development
• Regular Build
• Regular Testing
• Regular Deployment
• Regular Delivery/ Run
• Regular Monitoring and Feedback

This is one of the most confusing topics now a days as data is the most useable element of organizations now a day.

I won’t be going into details as we have discussed each as a separate topic but will only try to elaborate the difference between both.

Below is the high-level difference between both the practices.

• DevOps focuses on development by software developers, quality, and operations whereas DataOps focuses on source and target integrations, data transformation from data engineers, data quality and security.
• DevOps makes sure there is no hindrance in environment for HIGH QUALITY development cycle whereas DataOps makes sure, end users get HIGH QUALITY data, trusted data, ready to use data and actionable KPIs.
• DevOps make sure, all IT team members, operational team members, business team members are aligned to SPEED-UP the process after every sprint where DataOps makes sure, IT team members, development team members and business team members roles and responsibilities are aligned for FASTER collaborative work across teams.
• DevOps make sure, automation of all the servers management, versions control, configurations, testing, release management, etc. are AUTOMATED whereas DataOps makes sure, data acquisition, integration, metadata management, self-service, data governance, cross cloud, etc. are all in place and working in AUTOMATED fashion.
• DevOps works with stakeholders for real-time feedback after every sprint for FEEDBACK-ORIENTED optimization whereas DataOps works with stakeholders to get regular insights from new data flowing-in in real-time or in batch for RESULT-ORIENTED optimization.
• DevOps goes for regular implementations for QUICK recovery to cater failures in agile mode whereas DataOps goes for regular implementations to QUICKLY fix data quality and minimum metadata discovering in agile mode.

MLOps is also from the same family of DevOps, DevSecOps, DataOps, etc. Just like DataOps has a focus on data, MLOps focuses on Machine Learning Model’s end to end journey.

As the name says itself, MLOps (Machine Learning Operations) means from the availability of raw data till the ML model is deployed in PROD, this whole journey is called MLOps.

Please note, CI/ CD is a very critical part of MLOps as in Machine Learning world, 90% models do not go to Production. Most of the models are created, trained, and decommissioned due to N number of reasons. CI/CD plays a vital role of moving models from development to training area again and again and once confirmed by the owner, moved to Prod.

As mentioned, MLOps was extracted from the concept of DevOps, it follows the same pattern but has a different agenda.

The key phases of MLOps are: [These are taken from Best Practice on Databricks, Google and few other places.]

There is a very thin line between ModelOps and MLOps. ModelOps is the next stage of MLOps where focus is more towards automating, retraining, and maturing models on its own.

MLOps mainly focuses on model management, model monitoring and CI/ CD whereas ModelOps mainly focuses on addressing and sharing AI performance to the stakeholders without the need of Data Science and Machine Learning Team members.

Few differences between ModelOps and MLOps.

• MLOps focuses on ML models operationalization whereas ModelOps focuses on decision models and AI operationalization.
• MLOps focuses on continuous cycle of development, implementation, and monitoring of models whereas ModelOps focuses on life cycle management and governance of models.
• MLOps focuses on creating models for AI enabled applications whereas ModelOps focuses on AI transparency and usage of KPIs by business users.

In nutshell, ModelOps supports AI capabilities where involvement of Data Science become less and less as Models run on their own, retrain on their own, deploy on their own, extract business KPIs on their own etc.

AIOps was initiated by Gartner in 2016. It supports DevOps, DataOps, MLOps, DLOps and ITOps together.

AIOps came to surface after the inception of Big Data era where huge data started to flow in, where DevOps, DataOps, MLOps, DLOps rather every kind of operational tasks had tripled. And as we know everyone has recently realized the importance of Data so now everyone wants to progress in data ignoring the fact that automation requires processes and governance etc. That is why AIOps came into picture where everything is handled by the system itself.

AIOps is about assisting DevOps, DataOps, MLOps and ITOps etc., to work seamlessly. AIOps uses algorithms to make things happen like preventing outages, maintaining uptime and assure continuous services. Unlike DevOps, MLOps and ITOps where human interactions are still required, here AIOps makes sure everything runs on its own.

As mentioned, AIOps handles end to end operations on it owns, below are few stages.

Data Lifecycle Management (DLM) is mostly confused with Data Supply Chain which is not 100% correct.

Data Lifecycle Management is about the process from where data was generated till it was removed from the systems whereas Data Supply Chain is about where data come from and where it goes.

There are many sets of steps for Data Lifecycle management, we will be looking into the most commonly used.

Steps in Data Lifecycle Management (DLM)

• Data Generation means the channel where data is getting generated e.g., mobile application, CRM, ERP, HRM, CCTV, Social-Media, Audio(s), Emails, Word document, PDF, Images, Scans etc.
• Data Processing means, what are the business rules based on which raw data is transformed in required format which can be via ETL or ELT or via Data Mining or Data Science Machine Learning Modelling etc.
• Data Reconciliation, Presentation and Utilization means how transformed data is matched with raw data then how it’s been shown on Reports and Dashboard, and lastly for what purpose it has been used.
• Data Storage Lifespan and Extraction means, for how long the data will be kept and what are the means to extract it which can be manual or can be via automated tools.
• Data Removal means how data has been removed and based on which criteria(s).