Knowledge Hub

Data Imputation is majorly used in Data Science domain.

‘Imputation means, fill missing data.’

Data Imputation normally comes into play so business rules can be imposed on all data rather on subsets of data. Before Machine Learning Model runs, data imputation is done by either missing data is filled up or the whole row is removed.

Missing value can create many kinds of issues e.g., during categorizing, classifying, clustering etc., if data is missing then results will not be 100%.

Generally, in numeric datasets, missing values are replaced with MEAN values. Having said that, there many ways to deal with missing data.

Few Imputations Techniques

• The first technique is where data is missing at random basis but before using it, it is completely removed from the dataset.
• The second technique is also where data is missing at random basis but here missing values are replaced referring to the other data points available with current dataset.
• Third technique is where data is not missing at random basis and mainly replaced using Mean/Median/Mode values.

Data Fusion, another term overlapping with many other already in-place terms.

‘Data Fusion is about pulling same nature of data from multiple source systems to co-relate with each other to produce one unified output.’

For example, data for a particular product can come from POS (point of sales), from shops, from online stores etc. At the same time, people can talk about that product on social media, in seminars, webinars etc. Fusion of all those data elements can produce valuable information about that product for sales and marketing teams.

Yes, looking into image you can make out same thing can be achieved in Data Warehouse, Data Mart, Data Lake, Data Lakehouse, Data Vault/ Business Vault etc., and you are not wrong. All above-mentioned Data Store can be used for Data Fusion.

In this era, social media has surpassed every medium to generate data. Social media is not only about writing messages, it also about sending audio or video messages and the moment audio and video (with voice) comes into picture, NLP comes into action.

‘NLP is about using techniques to understand the way humans communicate.’

Using NLP, code tries to convert audio into text and vice versa. Using NLP capabilities there are solutions which talk to humans, understand what they want and then reply to them. NLP also assists in performing actions e.g., a human can ask ‘I want a Chill Coke Zero’, the NLP will recognize this request and get a Cold Coke Drink for customer. At the same time, if Chill Coke Zero is out of stock, using Machine Learning and AI, it can reply to customer in Voice format that ‘At the moment, we are out of stock, would you like to go for Chill Pepsi Black’.

Steps of NLP

• Understand
• Analyze
• Manipulate
• General Human Language or Text

The concept of text and embedded vectorization in machine learning can be a bit confusing when compared to general data or text vectorization. Both approaches share the overarching goal of accelerating tasks, but they operate in different ways.

General Vectorization: The primary aim is to convert a single action on a single data element (SISD) into a single action on multiple data elements. This involves transforming the data in a way that allows for parallel processing, improving efficiency in handling multiple data elements simultaneously.

Text Vectorization in Machine Learning: In the context of machine learning, text vectorization specifically deals with converting textual or character data into numerical representations. Once the text or characters are converted into numbers, the embedding process comes into play. This process involves using machine learning and deep learning techniques to learn from these numerical representations, facilitating faster and more effective results.

Generative Artificial Intelligence (AI) has become a prominent trend in recent times, particularly emphasized by our enthusiastic Sales folks 😊. But let's pause for a moment, are you familiar with the concept of Synthetic Data? Surely, it's not new to you 😊. And for those with a tech-savvy mindset, the term "Generative AI" likely triggers thoughts of Generative Models or Statistical Models, perhaps with a knowing smile.

Indeed, the application of generative and statistical models has been around for decades. In fact, the idea of Generative AI isn’t new at all, it dates back to as early as 1950, when Alan Turing proposed machines that could exhibit intelligent behavior.

Today, however, we're witnessing a resurgence of interest, largely due to the accessibility and power of tools like OpenAI’s ChatGPT, Google Gemini, Anthropic’s Claude, Meta's LLaMA, and Stability AI's Stable Diffusion. These tools represent a leap forward in putting generative capabilities into the hands of everyday users, generating text, images, audio, and even code with natural ease.

The term "Large Language Model" (LLM) generally refers to advanced artificial intelligence models designed to understand and generate human-like language. These models are typically trained on massive amounts of diverse text data and leverage deep learning techniques to capture the complexities of language. One of the notable examples of an LLM is GPT (Generative Pre-trained Transformer) like GPT-3 for ChatGPT in current era.

Here are key characteristics and aspects associated with Large Language Models:

• Training on Extensive Datasets: LLMs are trained on large and diverse datasets, which often include a wide range of text sources such as books, articles, websites, and other textual content from the internet.
• Pre-training and Fine-tuning: LLMs go through a pre-training phase where they learn to predict the next word in a sentence or fill in missing words. After pre-training, they can be fine-tuned on specific tasks or domains to enhance their performance in particular areas.
• Transformer Architecture: Many LLMs, including GPT models, use a transformer architecture. Transformers enable the models to capture long-range dependencies and contextual information in the data efficiently.
• Natural Language Understanding: LLMs excel in natural language understanding, allowing them to comprehend the meaning, context, and nuances of human language. This includes tasks such as language translation, summarization, question answering, and more.
• Generative Capabilities: One of the distinctive features of LLMs are their generative ability. They can generate coherent and contextually applicable textual content primarily based totally on a given activate or input.
• Applications: LLMs have found applications in various domains, including chatbots, language translation, content generation, summarization, sentiment analysis, and more.
• GPT Series: The GPT (Generative Pre-trained Transformer) series, developed by OpenAI, is a notable example of LLMs. GPT-3, the third iteration, is one of the largest and most powerful language models to date.

We have already gone through Analysis and Analytics. Analytics refers to the systematic analysis of data or statistics to uncover meaningful patterns, insights, and trends. There are several types of analytics, each serving specific purposes and addressing different aspects of data analysis.

The major types of analytics include:

• Descriptive Analytics: It describes what has happened in the past, providing a historical overview. For example, summarizing sales data to understand total revenue for a specific period.
• Diagnostic Analytics: It focuses on why a particular event or trend occurred by identifying its root causes. For example, investigating a drop in website traffic to determine the reasons behind the decline.
• Predictive Analytics: It predicts future outcomes or trends based on historical data and statistical algorithms. For example, using machine learning models to predict customer churn or stock prices.
• Prescriptive Analytics: It recommends actions to optimize outcomes based on predictive analytics results. For example, suggesting marketing strategies to increase customer retention based on predictive insights.
• Other types of analytics include:
• Text Analytics (Text Mining): It analyzes unstructured text data to extract insights, sentiment, or patterns. For example, analyzing customer reviews to understand sentiment and identify common themes.
• Spatial Analytics: It analyzes geographic or location-based data to uncover spatial patterns and relationships. For example, mapping crime rates in a city to identify high-crime areas.
• Social Media Analytics: It analyzes social media data to understand trends, sentiments, and user interactions. For example, monitoring brand mentions on social media to assess brand sentiment.
• Web Analytics: It analyzes website data to understand user behavior, track performance, and optimize user experience. For example, tracking page views, bounce rates, and click-through rates on a website.
• Business Intelligence (BI): It involves the use of tools and processes to analyze and visualize business data for decision-making. For example, creating dashboards to monitor key performance indicators (KPIs) in real-time.
• Big Data Analytics: It analyzes large and complex datasets to uncover hidden patterns and insights. For example, analyzing vast amounts of sensor data for predictive maintenance in manufacturing.
• Customer Analytics: It analyzes customer data to understand behavior, preferences, and trends. For example, creating customer segmentation based on purchasing patterns.

MRKL stands for Modular Reasoning, Knowledge, and Language. This neuro-symbolic architecture is designed to address complex problems by combining elements of neural computation (specifically Large Language Models, or LLMs) with symbolic computation tools such as calculators. (ref: LLM is separate topic)

Neuro-Symbolic Architecture implies that the system combines aspects of neural computation (associated with LLMs) and symbolic computation. Neural computation involves processing information in a way inspired by the human brain, while symbolic computation involves manipulating symbols to perform operations.

LLMs (Large Language Models) are sophisticated artificial intelligence models trained on extensive text data. They surpass in recognizing and producing human-like dialect.

External Tools Like Calculators in addition to neural computation, the system incorporates external symbolic computation tools, specifically mentioning calculators. This suggests a hybrid approach where both neural and symbolic methods are employed.

Just like Row is the physical content in a Relational Database, Knowledge Graph (KG) is the physical content in a Graph Database.

Knowledge Graph (KG) integrates closely with Graph Theory, MetaGraph, Taxonomies, and Ontologies, acting as a unifying framework that bridges these concepts.

Let’s decode…..

A Knowledge Graph is a structured representation of knowledge, where entities (nodes) and their relationships (edges) are stored and queried in a graph format.

In a relational database, the schema (tables, columns, constraints) is empty until you populate it with rows (data). Similarly, the Knowledge Graph is the populated version of Graph Theory (structure), MetaGraph (metadata), Taxonomies (classification), and Ontologies (rules).

In Relational Database, Table Orders with columns OrderID, CustomerID, OrderDate having Rows with actual order records.

In Knowledge Graph, Nodes represent Customers and Orders; edges represent Purchases in this way i.e., (Customer: Alice) -[:Purchases {Date: '2023-12-12'}]-> (Order: #12345)

The widespread adoption of large language models (LLMs) has led to increased expectations for factual accuracy. However, LLMs often generate incorrect answers with confidence. A notable example is when users query general-purpose models like OpenAI’s “gpt-4o” or DeepSeek’s “DeepSeek-V3” about recent events or niche topics; they may decide not to respond or provide fabricated yet convincing responses. The leading industry solution to mitigate this issue is Retrieval-Augmented Generation (RAG), introduced in Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (Lewis et al., 2020). RAG enhances LLMs by integrating external retrieval, ensuring more reliable and factual outputs while improving response contextuality and robustness.

Just like how a human is more likely to give a wrong answer when lacking information, AI models are more likely to make mistakes and hallucinate when they are missing context. For a given application, the model’s instructions are common to all queries, whereas context is specific to each query. By incorporating retrieval mechanisms, RAG enables LLMs to interact with real-time information, improving applications in search engines, customer support, academic research, and business intelligence. Unlike traditional search engines, which rely on keyword-based ranking, RAG synthesizes knowledge dynamically, allowing AI-driven solutions to be more adaptable and responsive. This chapter explores RAG’s framework, applications, implementation, security considerations, and future trends.