Knowledge Hub

Entity Recognition, commonly known as Named Entity Recognition (NER), is a technique used to identify and extract meaningful entities from unstructured text. It is also referred to as entity identification or entity extraction. While entity recognition has long been appreciated in data modeling, particularly during conceptual modeling when identifying core business entities, it plays an equally critical role in the data science domain.

In traditional structured systems, entities such as Customer, Product, Order, or Invoice are clearly defined within database schemas. However, modern organizations generate enormous volumes of unstructured data from social media posts, emails, PDF files, Word documents, comment columns, chat transcripts, and even video or audio content converted into JSON or text formats. For data modelers, this unstructured layer represents one of the greatest challenges when designing robust data models.

Causality Data in the data world refers to information that helps establish a cause-and-effect relationship between different variables or events. In other words, it involves understanding the influence or impact that changes in one variable have on another.

Causality is a fundamental concept in statistics and data analysis, especially when trying to make sense of complex systems or phenomena. Establishing causal relationships is important for drawing meaningful conclusions and making informed decisions. However, it is essential to word that correlation does now no longer imply causation. While two variables may be correlated, it doesn't necessarily mean that one causes the other.

In the context of data science and analytics, establishing causality often involves advanced statistical methods and experimentation.

Here are a few key points related to causality data:

• Experimental Design: Controlled experiments, where researchers manipulate one variable while keeping others constant, are often used to establish causality. This allows for the isolation of the variable of interest and helps draw conclusions about its impact.
• Observational Studies: In cases where controlled experiments are not feasible, observational studies are conducted to analyze existing data. However, establishing causality in observational studies can be more challenging due to potential confounding factors.
• Counterfactuals: Causality often involves comparing what happened to what might have happened in the absence of a particular intervention or change. This comparison is known as a counterfactual analysis.
• Temporal Order: Causality requires that the cause precedes the effect in time. Observing a temporal order between variables is an important aspect of establishing causation.
• Statistical Techniques: Various statistical techniques, such as regression analysis, propensity score matching, and instrumental variable analysis, are employed to analyze data and infer causal relationships.

Inference Data refers to the data produced by ML process of drawing conclusions, making predictions, or generalizing information based on the analysis of a features dataset. It is a crucial step in extracting meaningful insights from data and often involves the application of statistical methods.

Let's decode it.....

Imagine you want to know something about a big group of students, like all the students in your school. Instead of asking every single student (which could be too much work), you ask a smaller group. This smaller group is like a sample. In simpler terms, inference is like making smart guesses about everyone by looking at a few representative friends first. It's a bit like saying, "I can figure out what most people like, by asking a few of them because they're probably a lot like everyone else."

Inference data is a process of drawing meaningful conclusions about a larger dataset based on observed data, and it is a fundamental aspect of statistical reasoning and data analysis in various domains.

Data Labelling is about tagging something which is not identifiable coming from source. The Labelling can be on known data and on unknown data as well.

Let’s try to understand the relationship among all these 3 terms i.e., Data Labelling, Data Classification and Data Clustering.

• Data Clustering group the data based on existing parameter and falls under Unsupervised Learning.
• Data Labelling is the process of tagging data based on some defined parameter.
• Data Classification group the data based on manually LABELLED data and falls under Supervised Learning.

Since the era of unstructured dataset, data is coming in many formats. There are specialized tools which can tag or label the unstructured data set so it can become input for Machine Learning (ML) Models. There are also certain libraries used in ML Models to label unstructured data set which can then become input of another ML Model for Supervised Learning.

Data Annotation and Data Labelling works together to prepare dataset for Machine Learning Models in Data Science practice.

‘Data Annotation is the technique of doing Data Labelling.’

Data Annotation is a complete process with many steps to curate, whereas Data Labelling is ONE of the many steps in it.

Data Labelling is about tagging the data for training e.g., machine learning models whereas Data Annotation is the technique for tagging the data so ML models can reuse it as well. Data Labelling tags are used by ML models for future pattern recognitions whereas Data Annotation is about recognizing the data via a process.

Just like Data Labelling, Data Annotation is also for supervised machine learning to prepare data as an input and to store it for future reference. Not to mention, there are STILL many organizations doing manual Data Labelling. Data Annotation is about the process so Data Labelling can be automated for accuracy, eliminating human intervention. Bad Data Labelling can impact the outcome of applications.

This term is normally known by the Security department and is tagged with Hackers.

‘Data Scraping is also known as Web Scraping.’

The minor difference between both terms is Data scraping means searching the location of the data and then extracting it. In other words, it does not copy and paste rather extract data in a precise and accurate manner whereas Web scraping involves extracting exactly everything from web pages.

Data Scraping or Web Scraping is the process of importing information from a website into a spreadsheet or local file saved on your computer.

Popular uses of data scraping include:

• Research for web content/business intelligence.
• Pricing for travel booker sites/price comparison sites
• Finding sales leads/conducting market research by crawling public data sources (e.g., Yell and Twitter)
• Sending product data from an e-commerce site to another online vendor (e.g., Google Shopping)

It allows us to take unstructured, scattered data from multiple sources and collect it in one place and make it structured for better use.

MLOps is also from the same family of DevOps, DevSecOps, DataOps, etc. Just like DataOps has a focus on data, MLOps focuses on Machine Learning Model’s end to end journey.

As the name says itself, MLOps (Machine Learning Operations) means from the availability of raw data till the ML model is deployed in PROD, this whole journey is called MLOps.

Please note, CI/ CD is a very critical part of MLOps as in Machine Learning world, 90% models do not go to Production. Most of the models are created, trained, and decommissioned due to N number of reasons. CI/CD plays a vital role of moving models from development to training area again and again and once confirmed by the owner, moved to Prod.

As mentioned, MLOps was extracted from the concept of DevOps, it follows the same pattern but has a different agenda.

The key phases of MLOps are: [These are taken from Best Practice on Databricks, Google and few other places.]

There is a very thin line between ModelOps and MLOps. ModelOps is the next stage of MLOps where focus is more towards automating, retraining, and maturing models on its own.

MLOps mainly focuses on model management, model monitoring and CI/ CD whereas ModelOps mainly focuses on addressing and sharing AI performance to the stakeholders without the need of Data Science and Machine Learning Team members.

Few differences between ModelOps and MLOps.

• MLOps focuses on ML models operationalization whereas ModelOps focuses on decision models and AI operationalization.
• MLOps focuses on continuous cycle of development, implementation, and monitoring of models whereas ModelOps focuses on life cycle management and governance of models.
• MLOps focuses on creating models for AI enabled applications whereas ModelOps focuses on AI transparency and usage of KPIs by business users.

In nutshell, ModelOps supports AI capabilities where involvement of Data Science become less and less as Models run on their own, retrain on their own, deploy on their own, extract business KPIs on their own etc.

AIOps was initiated by Gartner in 2016. It supports DevOps, DataOps, MLOps, DLOps and ITOps together.

AIOps came to surface after the inception of Big Data era where huge data started to flow in, where DevOps, DataOps, MLOps, DLOps rather every kind of operational tasks had tripled. And as we know everyone has recently realized the importance of Data so now everyone wants to progress in data ignoring the fact that automation requires processes and governance etc. That is why AIOps came into picture where everything is handled by the system itself.

AIOps is about assisting DevOps, DataOps, MLOps and ITOps etc., to work seamlessly. AIOps uses algorithms to make things happen like preventing outages, maintaining uptime and assure continuous services. Unlike DevOps, MLOps and ITOps where human interactions are still required, here AIOps makes sure everything runs on its own.

As mentioned, AIOps handles end to end operations on it owns, below are few stages.

To understand Data Algorithms, let's quickly know what an algorithm is. It is a procedure consisting of multiple steps with a set of instructions for a desired result. It is independent of any language, in other words, an algorithm can be implemented in any language.

Algorithm Characteristics

• Unambiguous: It should be clear and unambiguous. Each of its steps, and its inputs/outputs should be clear and must lead to only one meaning.
• Input: It should have 0 or more well-defined inputs.
• Output: It should have 1 or more well-defined outputs and should match the desired output.
• Finiteness: It must terminate after a finite number of steps.
• Feasibility: It should be feasible with the available resources.
• Independent: It should have step-by-step directions, which should be independent of any programming code.

Now from a data point-of-view, how the algorithm behaves.

• Search: Algorithm to search an item in a data structure.
• Sort: Algorithm to sort items in a certain order.
• Insert: Algorithm to insert an item in a data structure.
• Update: Algorithm to update an existing item in a data structure.
• Delete: Algorithm to delete an existing item from a data structure.