Knowledge Hub

This is one of the most confusing topics now a days as data is the most useable element of organizations now a day.

I won’t be going into details as we have discussed each as a separate topic but will only try to elaborate the difference between both.

Below is the high-level difference between both the practices.

• DevOps focuses on development by software developers, quality, and operations whereas DataOps focuses on source and target integrations, data transformation from data engineers, data quality and security.
• DevOps makes sure there is no hindrance in environment for HIGH QUALITY development cycle whereas DataOps makes sure, end users get HIGH QUALITY data, trusted data, ready to use data and actionable KPIs.
• DevOps make sure, all IT team members, operational team members, business team members are aligned to SPEED-UP the process after every sprint where DataOps makes sure, IT team members, development team members and business team members roles and responsibilities are aligned for FASTER collaborative work across teams.
• DevOps make sure, automation of all the servers management, versions control, configurations, testing, release management, etc. are AUTOMATED whereas DataOps makes sure, data acquisition, integration, metadata management, self-service, data governance, cross cloud, etc. are all in place and working in AUTOMATED fashion.
• DevOps works with stakeholders for real-time feedback after every sprint for FEEDBACK-ORIENTED optimization whereas DataOps works with stakeholders to get regular insights from new data flowing-in in real-time or in batch for RESULT-ORIENTED optimization.
• DevOps goes for regular implementations for QUICK recovery to cater failures in agile mode whereas DataOps goes for regular implementations to QUICKLY fix data quality and minimum metadata discovering in agile mode.

MLOps is also from the same family of DevOps, DevSecOps, DataOps, etc. Just like DataOps has a focus on data, MLOps focuses on Machine Learning Model’s end to end journey.

As the name says itself, MLOps (Machine Learning Operations) means from the availability of raw data till the ML model is deployed in PROD, this whole journey is called MLOps.

Please note, CI/ CD is a very critical part of MLOps as in Machine Learning world, 90% models do not go to Production. Most of the models are created, trained, and decommissioned due to N number of reasons. CI/CD plays a vital role of moving models from development to training area again and again and once confirmed by the owner, moved to Prod.

As mentioned, MLOps was extracted from the concept of DevOps, it follows the same pattern but has a different agenda.

The key phases of MLOps are: [These are taken from Best Practice on Databricks, Google and few other places.]

DLOps is like MLOps and belongs to Data Science domain.

Before we try to understand DLOps as compared to MLOps, one must understand the difference between Machine Learning and Deep Learning. (There is separate topic on difference between Machine Learning and Deep Learning)

‘It won’t be wrong to say, MACHINE LEARNING is for simpler and more structured & semi-structured use cases whereas DEEP LEARNING is for complex and unstructured use cases.’

Let’s try to understand with a simple example. If we need to find out how many cars, bikes or buses there are in structured or semi-structured datasets then it’s relatively easy as data is in rows and columns format. We can have the width, length, and height of each car, bike or bus so using MACHINE LEARNING models we can easily ready this dataset and produce expected output. But what if we give pictures to ML models to extract the same result, will it be possible? The answer is No as ML models normally work on structured or semi-structured datasets. To find out cars, bikes, or buses from pictures, DEEP LEARNING models are used which can recognize the objects in the pictures and find out which presents car or bike or a bus.

There is a very thin line between ModelOps and MLOps. ModelOps is the next stage of MLOps where focus is more towards automating, retraining, and maturing models on its own.

MLOps mainly focuses on model management, model monitoring and CI/ CD whereas ModelOps mainly focuses on addressing and sharing AI performance to the stakeholders without the need of Data Science and Machine Learning Team members.

Few differences between ModelOps and MLOps.

• MLOps focuses on ML models operationalization whereas ModelOps focuses on decision models and AI operationalization.
• MLOps focuses on continuous cycle of development, implementation, and monitoring of models whereas ModelOps focuses on life cycle management and governance of models.
• MLOps focuses on creating models for AI enabled applications whereas ModelOps focuses on AI transparency and usage of KPIs by business users.

In nutshell, ModelOps supports AI capabilities where involvement of Data Science become less and less as Models run on their own, retrain on their own, deploy on their own, extract business KPIs on their own etc.

With the introduction of the big data era, it became impossible to manually manage those.

‘ITOps focuses on the best in practice IT operations.’

A few main areas of focus for ITOps are.

• IT Service Improvement
• IT Service Strategy and Design
• IT Service Transition
• IT Service Operations
• There are many ITOps lifecycles followed by organizations like
• Maintain the HEALTH of the IT Infrastructure
• Visibility of the IT Infrastructure
• Automation of IT Services
• Optimization of IT Services

AIOps was initiated by Gartner in 2016. It supports DevOps, DataOps, MLOps, DLOps and ITOps together.

AIOps came to surface after the inception of Big Data era where huge data started to flow in, where DevOps, DataOps, MLOps, DLOps rather every kind of operational tasks had tripled. And as we know everyone has recently realized the importance of Data so now everyone wants to progress in data ignoring the fact that automation requires processes and governance etc. That is why AIOps came into picture where everything is handled by the system itself.

AIOps is about assisting DevOps, DataOps, MLOps and ITOps etc., to work seamlessly. AIOps uses algorithms to make things happen like preventing outages, maintaining uptime and assure continuous services. Unlike DevOps, MLOps and ITOps where human interactions are still required, here AIOps makes sure everything runs on its own.

As mentioned, AIOps handles end to end operations on it owns, below are few stages.

There are four types of instructions to pull data i.e., SISD, SIMD, MISD and MIMD.

• SISD: Single Instruction Stream and Single Data Stream
• SIMD: Single Instruction Stream and Multiple Data Stream
• MISD: Multiple Instruction Stream and Single Data Stream
• MIMD: Multiple Instruction Stream and Multiple Data Stream

Before we jump into understanding all above mentioned techniques, let’s first understand what is a Core in a machine or in a computer or in a server etc.?

A CPU Core is built of billions of microscopic transistors to process instructions passed to it. One instruction is handled by One core, if someone needs multitasking or multiple instruction in parallel, minimum 2 cores are required.

We know when we insert or load data in a computer or server or in a node, it gets saved in hard disk. We won’t go in to details how it is stored on the hard disk. We can imagine when we write something on paper, consider the same way, data is saved in a hard disk.

Data Lifecycle Management (DLM) is mostly confused with Data Supply Chain which is not 100% correct.

Data Lifecycle Management is about the process from where data was generated till it was removed from the systems whereas Data Supply Chain is about where data come from and where it goes.

There are many sets of steps for Data Lifecycle management, we will be looking into the most commonly used.

Steps in Data Lifecycle Management (DLM)

• Data Generation means the channel where data is getting generated e.g., mobile application, CRM, ERP, HRM, CCTV, Social-Media, Audio(s), Emails, Word document, PDF, Images, Scans etc.
• Data Processing means, what are the business rules based on which raw data is transformed in required format which can be via ETL or ELT or via Data Mining or Data Science Machine Learning Modelling etc.
• Data Reconciliation, Presentation and Utilization means how transformed data is matched with raw data then how it’s been shown on Reports and Dashboard, and lastly for what purpose it has been used.
• Data Storage Lifespan and Extraction means, for how long the data will be kept and what are the means to extract it which can be manual or can be via automated tools.
• Data Removal means how data has been removed and based on which criteria(s).

We all have heard about MPP (Massive Parallel Processing). In the current era, consumption and serving layer without MPP capability cannot survive.

MPP is mostly used to read and write (mostly insert) in parallel where more than 1 processor is involved.

In MPP, multiple processers are used to process queries in parallel which might be running in single node or across multiple nodes, in same location or multiple locations.

In a non-MPP mode when a query is executed its different pieces of code are executed one by one in a sequence. But in MPP mode when a query is executed its different pieces of code are executed in parallel on all the processors in one go. This increases the execution of queries 100 times faster than compared to one-by-one execution.

In the current era where data coming from social-media, audios, videos etc., are coming in such a velocity/ pace/ speed that without MPP, it will not be possible to write, process or read data in timely manner.

Data Skew Issue is one of the main hidden culprits for query performance 😊. It is normally referred to as Skewed data distribution which is used in Data Science domain.

Have you ever noticed, your query is correct, your joins are correct, your all indexes are enabled, you have high data quality, you have data standardization in place but still your query performance is not as expected? You noticed when you loaded new datasets, queries were flying but as time passed, the performance has gone down. You have added more CPU, more RAM, more nodes but NO performance is not like newly added data. It’s because of the Data Skew Issue.

Data Skew Issue is about when data is not distribution evenly. Uneven distribution of data at cluster level causes this issue. It heavily degrades the performance of the queries especially those with joins.

We all have faced a situation where query run exceptionally good till 99% and then hangs for 3-4 hours right lolz yeah, I can understand your pain 😊. Let me tell you the reason, it’s because of Data Skew Issue.