Knowledge Hub

Entity Recognition, commonly known as Named Entity Recognition (NER), is a technique used to identify and extract meaningful entities from unstructured text. It is also referred to as entity identification or entity extraction. While entity recognition has long been appreciated in data modeling, particularly during conceptual modeling when identifying core business entities, it plays an equally critical role in the data science domain.

In traditional structured systems, entities such as Customer, Product, Order, or Invoice are clearly defined within database schemas. However, modern organizations generate enormous volumes of unstructured data from social media posts, emails, PDF files, Word documents, comment columns, chat transcripts, and even video or audio content converted into JSON or text formats. For data modelers, this unstructured layer represents one of the greatest challenges when designing robust data models.

Causality Data in the data world refers to information that helps establish a cause-and-effect relationship between different variables or events. In other words, it involves understanding the influence or impact that changes in one variable have on another.

Causality is a fundamental concept in statistics and data analysis, especially when trying to make sense of complex systems or phenomena. Establishing causal relationships is important for drawing meaningful conclusions and making informed decisions. However, it is essential to word that correlation does now no longer imply causation. While two variables may be correlated, it doesn't necessarily mean that one causes the other.

In the context of data science and analytics, establishing causality often involves advanced statistical methods and experimentation.

Here are a few key points related to causality data:

• Experimental Design: Controlled experiments, where researchers manipulate one variable while keeping others constant, are often used to establish causality. This allows for the isolation of the variable of interest and helps draw conclusions about its impact.
• Observational Studies: In cases where controlled experiments are not feasible, observational studies are conducted to analyze existing data. However, establishing causality in observational studies can be more challenging due to potential confounding factors.
• Counterfactuals: Causality often involves comparing what happened to what might have happened in the absence of a particular intervention or change. This comparison is known as a counterfactual analysis.
• Temporal Order: Causality requires that the cause precedes the effect in time. Observing a temporal order between variables is an important aspect of establishing causation.
• Statistical Techniques: Various statistical techniques, such as regression analysis, propensity score matching, and instrumental variable analysis, are employed to analyze data and infer causal relationships.

Inference Data refers to the data produced by ML process of drawing conclusions, making predictions, or generalizing information based on the analysis of a features dataset. It is a crucial step in extracting meaningful insights from data and often involves the application of statistical methods.

Let's decode it.....

Imagine you want to know something about a big group of students, like all the students in your school. Instead of asking every single student (which could be too much work), you ask a smaller group. This smaller group is like a sample. In simpler terms, inference is like making smart guesses about everyone by looking at a few representative friends first. It's a bit like saying, "I can figure out what most people like, by asking a few of them because they're probably a lot like everyone else."

Inference data is a process of drawing meaningful conclusions about a larger dataset based on observed data, and it is a fundamental aspect of statistical reasoning and data analysis in various domains.

Knowledge Graph Modelling is the practice of representing information as a network of interconnected entities and relationships, enriched with semantic meaning. Instead of viewing data as isolated records stored in tables, this approach treats knowledge as a web of connections. Entities become nodes in a graph, and relationships between them become edges. The structure itself reflects how concepts are linked in the real world.

The primary goal of Knowledge Graph Modelling is to create a flexible and interconnected view of knowledge. By explicitly representing relationships, it becomes easier to integrate information from multiple sources, discover hidden patterns, and enable reasoning across domains. This modelling approach is especially valuable in environments where connections matter as much as individual data points.

A knowledge graph models the world as nodes and edges. Nodes represent entities such as Customer, Loan, or Product. Edges represent relationships such as applies for, owns, guarantees, or reviews. Unlike traditional relational models, where relationships are often embedded in foreign keys, the connections in a knowledge graph are first-class elements. They are visible, traversable, and central to the model.

AI / Machine Learning Feature Modelling is the discipline of designing, structuring, and governing features, the measurable variables used as inputs to machine learning models. It defines how features are created, catalogued, versioned, and reused so that model inputs remain consistent, explainable, and trustworthy across the enterprise.

The primary goal of Feature Modelling is to provide a standardized and governed framework for engineering and managing features. Instead of allowing every data scientist to independently create slightly different versions of the same metric, Feature Modelling ensures that features are defined once, documented clearly, and reused consistently across multiple AI use cases.

In practice, raw transactional or external data is rarely fed directly into machine learning algorithms. It is first transformed into structured features such as:

• Customer credit score
• Average repayment delay
• Loan utilization ratio
• Officer approval rate
• Customer income stability index

Please note, AAaaS has nothing to do with Cloud, it can be on On-Premises or On-Cloud. AAaaS is an environment setup for Data Scientists to play around with Machine Learning and Deep Learning Modelling.

Advance Analytics as a Service (AAaaS) is a new concept and still maturing, due to which it is still very confusing. Different companies are offering different approaches, so there is no one best approach.

Two approaches been followed.

1. AAaaS as PaaS: Modelling tools like Databricks, Azure AML etc., are available as pre-installed services (refer to topic: PaaS) where customers can provision these tools in few clicks and simply start developing their Machine Learning or Deep Learning Models to build their own solutions. Here, customers must have their own MLOps along with CI/CD.
2. AAaaS as SaaS: In this approach, Solution Providers hosts pre-built or OOTB Machine Learning and Deep Learning Models. These are offered to customers to consume only. Customers inject their own dataset for relevant data analysis and analytics. Here, customers must follow Solution Providers MLOps along with CI/CD.

‘Data Segmentation is about dividing dataset into multiple sections based on some requirements.’

Segments are defined based on many parameters e.g., in a business, segments are defined by accessibility, actionable, profitable etc.

For example, when we go into a mall, there is a segment/section for Food Court where multiple categories of food are available with different classification like expensive or cheap food.

Steps to improve Data Segmentation

• Enrich your data to increase data quality.
• Plan and replan your data segmentation criteria(s)
• Improve real time segmentation.
• Design segment based on customer prioritization and feedback.
• Integrate with as many data elements.

In business, what you can do by having Data Segmentation

• You can create deals based on customer buying behavior.
• You can improve your products arrangement.
• You can plan your marketing strategy.
• You can analyze customer feedback.

Data Labelling is about tagging something which is not identifiable coming from source. The Labelling can be on known data and on unknown data as well.

Let’s try to understand the relationship among all these 3 terms i.e., Data Labelling, Data Classification and Data Clustering.

• Data Clustering group the data based on existing parameter and falls under Unsupervised Learning.
• Data Labelling is the process of tagging data based on some defined parameter.
• Data Classification group the data based on manually LABELLED data and falls under Supervised Learning.

Since the era of unstructured dataset, data is coming in many formats. There are specialized tools which can tag or label the unstructured data set so it can become input for Machine Learning (ML) Models. There are also certain libraries used in ML Models to label unstructured data set which can then become input of another ML Model for Supervised Learning.

Data Annotation and Data Labelling works together to prepare dataset for Machine Learning Models in Data Science practice.

‘Data Annotation is the technique of doing Data Labelling.’

Data Annotation is a complete process with many steps to curate, whereas Data Labelling is ONE of the many steps in it.

Data Labelling is about tagging the data for training e.g., machine learning models whereas Data Annotation is the technique for tagging the data so ML models can reuse it as well. Data Labelling tags are used by ML models for future pattern recognitions whereas Data Annotation is about recognizing the data via a process.

Just like Data Labelling, Data Annotation is also for supervised machine learning to prepare data as an input and to store it for future reference. Not to mention, there are STILL many organizations doing manual Data Labelling. Data Annotation is about the process so Data Labelling can be automated for accuracy, eliminating human intervention. Bad Data Labelling can impact the outcome of applications.

Entropy term is normally used in Data Science domain.

• Where finding unexpected outcomes is the aim
• Where surprises are welcomed
• Where results are analyzed based on probability ratio, the lower the probability the better it is.
• Where informative information is something which wasn’t known

Types

• High Entropy means, more surprises, more unexpected values are considered more informative.
• Low Entropy means, less surprises, less unexpected values are considered less informative.

‘Entropy is also called as Extreme Disorder of values.’

Referring to the image, we can see at the starting point all the signs are MINUS, then in the middle there are 50/50 signs of PLUS & MINUS and right at the end all the signs are PLUS. At those extremes left, middle and right corners, the Entropy is at the lowest, so no surprises are expected so nothing much for Data Scientists to predict right. But in 2^nd, 4^th, and 5^th circles, it’s difficult to say how many PLUSES(s) and MINUS(s), if values are not visible, at this points Entropy is at the highest.