RAG(Retrieval-Augmented Generation)

The widespread adoption of large language models (LLMs) has led to increased expectations for factual accuracy. However, LLMs often generate incorrect answers with confidence. A notable example is when users query general-purpose models like OpenAI’s “gpt-4o” or DeepSeek’s “DeepSeek-V3” about recent events or niche topics; they may decide not to respond or provide fabricated yet convincing responses. The leading industry solution to mitigate this issue is Retrieval-Augmented Generation (RAG), introduced in Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (Lewis et al., 2020). RAG enhances LLMs by integrating external retrieval, ensuring more reliable and factual outputs while improving response contextuality and robustness.

Just like how a human is more likely to give a wrong answer when lacking information, AI models are more likely to make mistakes and hallucinate when they are missing context. For a given application, the model’s instructions are common to all queries, whereas context is specific to each query. By incorporating retrieval mechanisms, RAG enables LLMs to interact with real-time information, improving applications in search engines, customer support, academic research, and business intelligence. Unlike traditional search engines, which rely on keyword-based ranking, RAG synthesizes knowledge dynamically, allowing AI-driven solutions to be more adaptable and responsive. This chapter explores RAG’s framework, applications, implementation, security considerations, and future trends.

Understanding the RAG Framework

RAG systems integrate retrieval and generation capabilities, reducing hallucinations and improving factual consistency. This approach allows models to dynamically retrieve relevant data, ensuring responses are grounded in up-to-date, reliable information. Instead of solely relying on pre-trained knowledge, RAG fetches information from external sources such as databases, search engines, or internal company documents.

The retrieve-then-generate pattern was first introduced in Reading Wikipedia to Answer Open-Domain Questions (Chen et al., 2017). This method initially retrieved relevant Wikipedia pages before a model generated an answer using those documents. The term retrieval-augmented generation was later coined in Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (Lewis et al., 2020), formalizing RAG as a solution for knowledge-intensive tasks where static knowledge is insufficient.

RAG systems incorporate search capabilities in addition to generation capabilities. They can be seen as an improvement to generation systems because they reduce their hallucinations and improve their factuality. They also enable use cases of “chat with my data” that consumers and companies can use to ground an LLM on internal company data, or a specific data source of interest (e.g., chatting with a book). This also extends to search systems. More search engines are incorporating an LLM to summarize results or answer questions submitted to the search engine. Examples include Perplexity, Microsoft Bing AI, and Google Gemini.

For instance, if a user queries, “What is the size of the Infotainment Head Unit of Proton X70 2025?”, the model’s response will be significantly improved if it has access to the specifications of the Proton X70. RAG enables AI systems to construct context dynamically for each query rather than relying on static, predefined context. This dynamic approach is particularly beneficial in managing user data, as it ensures that personalized information is only included in relevant queries. In essence, constructing context in foundation models serves a similar purpose to feature engineering in classical machine learning, both aim to provide the necessary data for optimal processing.

While many assume that extending a model’s context length will eliminate the need for RAG, this is unlikely. The volume of available data continuously expands, and users frequently generate new information while rarely deleting old data. Thus, there will always be scenarios requiring context that exceeds a model’s inherent capacity.

Moreover, longer context windows do not necessarily translate to more effective context utilization. Additionally, increasing context size introduces computational overhead and latency. RAG mitigates these challenges by selectively retrieving only the most relevant information, optimizing both accuracy and efficiency.

RAG Architecture

A Retrieval-Augmented Generation (RAG) system consists of two fundamental components:

• Retriever: Identifies and extracts the most relevant information from external data sources, such as databases, APIs, or vector stores.
• Generator: Utilizes the retrieved information to generate a well-grounded, contextually relevant response.

This architecture follows a retrieve-then-generate approach, first introduced in Reading Wikipedia to Answer Open-Domain Questions (Chen et al., 2017). Instead of relying solely on a model’s internal knowledge, RAG dynamically integrates external data into its responses, significantly improving factual accuracy and reducing hallucinations. The approach was later formalized in Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (Lewis et al., 2020), which demonstrated how retrieval mechanisms enhance LLM outputs.

How RAG Works

• Retrieval Phase: When a user submits a query, the retriever searches external memory (e.g., vector databases, indexed documents, or APIs) for relevant information. This can be done using term-based retrieval methods (e.g., BM25), semantic retrieval (e.g., vector search), or hybrid retrieval.
• Context Construction: The retrieved documents or passages are formatted and added to the model's prompt, providing a tailored context specific to the user’s query.
• Generation Phase: The generative model processes the query along with the retrieved context to produce a coherent and well-informed response.
• Post-Processing: Some systems refine responses further by applying citation validation, summarization, or ranking mechanisms to ensure quality and accuracy.

Originally, RAG systems were trained with joint optimization of both retrieval and generation components. However, modern implementations typically train them separately, allowing for flexibility in selecting retrieval mechanisms and generative models. Fine-tuning both components end-to-end can enhance performance, particularly for domain-specific applications.

The success of a RAG system largely depends on retrieval quality. Organizations implementing RAG must consider factors like indexing strategies, document chunking, and retrieval precision to maximize effectiveness. By dynamically constructing query-specific context, RAG allows AI systems to remain up-to-date and scalable, overcoming limitations associated with static model parameters

Retrieval Optimization

Optimizing retrieval strategies is essential for improving the accuracy and efficiency of a RAG system. Several key techniques enhance document retrieval: chunking strategy, reranking, query rewriting, and contextual retrieval.

• Chunking Strategy

The effectiveness of retrieval depends on how data is indexed and structured. Chunking strategies play a critical role in segmenting documents for efficient retrieval.

• • Fixed-Length Chunking: Documents are divided into chunks of equal length based on a chosen unit, characters, words, sentences, or paragraphs. For example, a document can be split into segments of 2,048 characters or 512 words to maintain consistency.
  • Recursive Chunking: A hierarchical approach where documents are first split into sections, then paragraphs, and finally into sentences if necessary. This method reduces arbitrary segmentation and ensures logical grouping.
  • Domain-Specific Chunking: Certain document types require specialized chunking strategies. For instance, programming documentation may be split by function definitions, while Q&A documents may be divided by question-answer pairs.
  • Token-Based Chunking: Texts can be split based on token limits imposed by language models. For example, when using Llama 3, documents are tokenized according to its tokenizer, ensuring compatibility with the model’s context length.

A well-structured chunking strategy prevents critical information loss while ensuring efficient indexing and retrieval. Additionally, incorporating slight overlaps between chunks ensures that context is preserved and reduces fragmentation.

• Reranking

Reranking refines the results retrieved by the system to prioritize the most relevant information. This is particularly beneficial when dealing with large datasets or when the model’s context window is limited.

• • Precision Enhancement: An initial retrieval fetches a broad set of relevant documents, which are then reordered based on their relevance scores.
  • Time-Based Reranking: More recent data may be given higher priority, which is useful in applications like news aggregation, financial analysis, or AI-powered email assistants.
  • Contextual Relevance: Unlike traditional search, where exact positioning in a ranking is crucial, RAG models benefit from context-aware ranking that ensures documents with high informational value are retained in the model’s context.
• Query Rewriting

Query rewriting, also known as query reformulation or normalization, enhances retrieval by refining user inputs to ensure more effective document matching.

• Example Scenario:
  • User Input: “How about Mustafa?”
  • Problem: This question lacks sufficient context.
  • Rewritten Query: “When was the last time Mustafa made a purchase from us?”

By clarifying ambiguous queries, the system ensures accurate retrieval results. Traditional search engines use heuristic-based rewriting, whereas AI-powered systems can leverage LLM's to generate precise query refinements dynamically. This process becomes even more complex when requiring entity resolution, such as identifying individuals or cross-referencing external databases.

• Contextual Retrieval

Contextual retrieval enhances document retrieval by augmenting data chunks with relevant metadata, improving precision.

• • Metadata Augmentation: Adding structured metadata like keywords, tags, or extracted entities (e.g., error codes, product names) improves retrieval accuracy.
  • Question-Based Augmentation: Associating documents with common queries improves response relevance. For instance, a customer support document on password resets may be indexed with multiple variations such as “How to reset my password?” or “I forgot my password.”
  • Contextual Chunking: If a document is segmented, the system ensures each chunk retains references to its original document, reducing the likelihood of losing critical information. AI-generated summaries can also be prepended to each chunk to improve retrievability.

Applications of RAG

• Search Engines

RAG-powered search engines go beyond traditional keyword-based ranking. They summarize information from multiple sources, reducing the need for users to manually sift through search results. Examples include Perplexity, Microsoft Bing AI, and Google Gemini, where RAG enhances content discovery and personalized search recommendations.

• Customer Support

Instead of presenting users with a list of potentially relevant FAQ pages, a RAG-powered chatbot can retrieve and summarize company policies or support articles to provide direct, contextual responses. This helps reduce response time and ensures that customers receive precise, relevant information in real time.

• Enterprise Solutions

Industries such as finance and healthcare require highly accurate and compliant AI systems. RAG ensures that models retrieve data only from verified internal sources, mitigating risks associated with misinformation and regulatory non-compliance. For instance, financial analysts can leverage RAG to extract and analyze corporate earnings reports, while healthcare professionals can retrieve up-to-date clinical guidelines.

• Academic & Legal Research

RAG enhances research workflows by retrieving and synthesizing findings from multiple documents, aiding researchers and legal professionals in answering complex queries efficiently. Legal professionals can use RAG to analyze case law, statutes, and regulations, while academic researchers can quickly survey vast literature databases.

API vs. Local Models in RAG

When implementing Retrieval-Augmented Generation (RAG) systems, organizations must choose between using cloud-based APIs or running local models. Each approach has distinct advantages and trade-offs, depending on factors like computational resources, security requirements, and performance needs.

Cloud API Models

Cloud-based APIs provide seamless access to state-of-the-art language models without the need for extensive infrastructure. Providers like OpenAI and DeepSeek offer scalable, high-performance models that can be integrated via API calls, reducing the computational burden on local systems.

Advantages of Cloud APIs:

• Scalability: APIs allow businesses to scale AI applications efficiently without investing in expensive hardware.
• Continuous Updates: Cloud providers frequently update models, ensuring access to the latest advancements in AI.
• Lower Maintenance Costs: Organizations avoid managing on-premise hardware and optimizing models manually.
• High-Performance Models: Models like OpenAI’s GPT-4o, o1, o3 and DeepSeek’s LLMs provide cutting-edge natural language processing capabilities, enabling complex AI-driven applications.

Example: OpenAI API

A customer service chatbot built on OpenAI’s GPT-4o API can provide real-time, high-quality responses to user queries while dynamically retrieving relevant documents using RAG. The cloud-based approach ensures that the chatbot remains up-to-date and can handle fluctuating demand without latency issues.

Local Models

Local models run on an organization’s own hardware, offering greater control over data privacy and security. Open-source models like Ollama and DeepSeek LLM enable companies to deploy powerful AI capabilities while maintaining full ownership of their data.

Advantages of Local Models:

• Enhanced Security & Privacy: Local models eliminate the need to send sensitive data over external networks.
• Offline Accessibility: Organizations can run AI models without requiring internet connectivity, crucial for edge computing or restricted environments.
• Customization & Control: Fine-tuning and optimization can be performed in-house, adapting models to specific use cases.
• Lower Long-Term Costs: While initial infrastructure investment is required, local models can reduce API usage costs over time.

Example: Ollama for Local AI

A financial institution implementing Ollama to power an internal RAG-based document assistant ensures that all client data remains within secure, on-premises servers. This setup allows the assistant to retrieve and generate responses based on proprietary financial reports without exposing sensitive information to external cloud services.

Choosing Between API and Local Models

The decision between API-based and local models depends on specific business needs:

• Use Cloud APIs when prioritizing performance, scalability, and ease of deployment (e.g., customer support chatbots, real-time AI services).
• Use Local Models when prioritizing data security, offline access, and model customization (e.g., healthcare, finance, and government applications).

By balancing these considerations, organizations can optimize their RAG implementations for both efficiency and security.

Advanced RAG Techniques

Retrieval-Augmented Generation (RAG) enhances the capabilities of Large Language Models (LLMs) by integrating external knowledge sources into the generation process. Advanced RAG techniques have been developed to improve the efficiency, accuracy, and relevance of information retrieval and content generation. Here are some notable methods:

• Dense Retrieval

This technique involves converting both queries and documents into dense vector representations that capture semantic meanings. By computing the similarity between these vectors, the system retrieves documents that are semantically relevant, even if they don't share exact keywords with the query. This approach enhances the retrieval of pertinent information, especially in scenarios where exact keyword matches are insufficient.

• Reranking

After an initial retrieval phase, reranking methods reorder the retrieved documents to prioritize the most relevant ones. Advanced reranking models, such as cross-encoders, jointly process the query and each document to assess relevance more accurately. This step ensures that the most pertinent information is considered during the generation phase.

• Multi-Step Reasoning

Incorporating multi-step reasoning allows the system to handle complex queries that require synthesizing information from multiple sources. By iteratively retrieving and generating information, the model can construct coherent and contextually accurate responses to intricate questions.

• Query Expansion

This technique involves reformulating the user's query to include additional terms or phrases that capture the underlying intent more effectively. By expanding the query, the retrieval system can access a broader range of relevant documents, improving the chances of retrieving pertinent information.

• Knowledge Graph-Augmented Retrieval

Integrating knowledge graphs into the retrieval process enhances the system's ability to understand and utilize structured information. Knowledge graphs provide a network of entities and their relationships, enabling the retrieval system to disambiguate queries, enrich context, and infer user intent more effectively.

• Modular RAG Framework

Adopting a modular approach involves decomposing the RAG system into distinct components, such as query rewriting, retrieval, reranking, and generation. This modularity allows for independent optimization and customization of each component, facilitating experimentation with different configurations and techniques to enhance overall system performance.

• Agentic RAG & Hybrid Search

Agentic RAG extends traditional RAG systems by incorporating decision-making capabilities into the retrieval and generation process. Unlike standard RAG, which passively retrieves and augments responses based on predefined retrieval steps, Agentic RAG actively determines how and where to search for information, dynamically adjusting its strategy based on the query and available data sources. Combining keyword-based search with semantic retrieval enhances accuracy, especially for ambiguous or complex queries, ensuring high-precision outputs.

• Caching & Performance Optimizations

Caching frequently retrieved results reduces redundancy and speeds up response times. Optimized index structures improve retrieval efficiency, reducing computational load.

Security Considerations in Retrieval-Augmented Generation (RAG)

The integration of the Retrieval-Augmented Generation (RAG) pattern in Generative AI (GenAI) applications introduces intricate interactions between data retrieval, processing, and generation components. This complexity necessitates a comprehensive security strategy to ensure the integrity, confidentiality, and availability of the system. The following are four critical security considerations that must be addressed:

1. Prevent Embedding of Personally Identifiable Information (PII) or Sensitive Data in Vector Databases

Vector databases serve as essential repositories within the RAG architecture, often containing vast amounts of structured knowledge. Ensuring that PII and other sensitive data are not embedded in these databases is paramount.

Importance: Storing sensitive information in vector databases increases the risk of unauthorized access or data breaches, potentially leading to privacy violations and regulatory non-compliance.

Mitigation Strategies:

• Perform rigorous data classification and sanitization to identify and exclude PII or sensitive information before vectorization.
• Establish robust data governance policies for handling and storing sensitive information within the vector database.
• Employ anonymization or pseudonymization techniques to de-identify data, rendering it untraceable to individual users.
• Provide users with the ability to opt out of having their data used in AI systems.

2. Implement Strict Access Controls for Vector Databases

Given the nature of similarity search in vector databases, they are particularly susceptible to unauthorized access and potential data exposure.

Importance: Unauthorized access can expose stored data as well as structural relationships between data points, potentially facilitating inferential attacks and data leakage.

Mitigation Strategies:

• Enforce Role-Based Access Control (RBAC) to restrict access strictly to authorized personnel, adhering to the principles of least privilege and need-to-know.
• Utilize encryption for data both in transit and at rest to protect against unauthorized interception or breaches.
• Conduct continuous monitoring and auditing of access logs to detect and mitigate unauthorized access attempts.
• Implement network segmentation and tenant isolation to further secure the vector database environment.

3. Secure Access to Large Language Model (LLM) APIs

Protecting access to LLM APIs is critical for preserving the integrity and confidentiality of the generation process.

Importance: Unauthorized access to LLM APIs can result in misuse, manipulation, or extraction of proprietary information, particularly if the model has been fine-tuned using confidential data.

Mitigation Strategies:

• Deploy strong authentication mechanisms, including API keys, OAuth tokens, or client certificates, to regulate API access.
• Enforce Multi-Factor Authentication (MFA) to enhance security measures.
• Apply rate limiting and usage quotas to prevent API abuse and excessive consumption.
• Monitor API interactions and establish alert mechanisms to detect and respond to anomalous or unauthorized access patterns.

4. Validate Generated Data Before Delivering Responses to Clients

Ensuring that generated responses meet quality, relevance, and security standards is essential to maintaining trust and mitigating risks.

Importance: Without proper validation, AI-generated responses may include inaccuracies, inappropriate content, or even malicious code injections, leading to misinformation or security vulnerabilities.

Mitigation Strategies:

• Implement content validation frameworks that review and filter AI-generated outputs based on predefined rules, including the detection of inappropriate language or potential security threats.
• Apply contextual validation to confirm that generated responses align with user queries and do not inadvertently disclose sensitive or unintended information.
• Incorporate a "human-in-the-loop" approach for reviewing critical or sensitive outputs to ensure adherence to quality and ethical standards.

Employ cross-validation techniques by leveraging multiple LLMs or non-LLM-based verification methods to enhance reliability.

RAG Beyond Text

While traditional Retrieval-Augmented Generation (RAG) systems primarily rely on text-based external data sources, they can also integrate multimodal and structured tabular data to enhance retrieval quality and contextual relevance.

Multimodal RAG

Multimodal RAG extends the capabilities of text-based retrieval by incorporating additional data modalities such as images, videos, and audio. This approach is particularly useful for applications requiring cross-modal information retrieval.

For example, when answering the query “What is the color of the temple in the movie Kuching Kungfu?”, a RAG system can retrieve both textual descriptions and relevant images, enhancing the accuracy of the response. If an image contains metadata such as titles, tags, or captions, the system can use these attributes for retrieval. Additionally, image similarity search can be employed using multimodal embedding models like CLIP (Radford et al., 2021), which enables retrieval based on the semantic alignment of text and visual data.

• Multimodal Retrieval Process
  • Generate embeddings for both textual and image data using a multimodal embedding model (e.g., CLIP).
  • Convert the user’s query into an embedding.
  • Perform a similarity search in a vector database to retrieve the most relevant images or texts.

By leveraging multimodal retrieval, RAG systems can provide more comprehensive responses in applications such as visual question answering, medical diagnostics, and content recommendation systems.

RAG with Tabular Data

In addition to unstructured data like text and images, RAG systems can also process structured data stored in tables, such as relational databases, spreadsheets, and enterprise records. Unlike traditional retrieval methods, where documents are fetched based on text similarity, tabular data retrieval often requires structured queries.

Example Use Case: E-Commerce Sales Analysis

Consider an e-commerce platform, Kuching Mart, specializing in cat fashion. The company maintains an order database named Sales, containing transaction records. If a user queries, “How many units of Mini Mimi were sold in the last 7 days?”, a RAG system must retrieve structured data from the database and generate a relevant response.

Structured Retrieval Workflow

Text-to-SQL Conversion: The system interprets the user query and formulates an SQL query to retrieve relevant data.

SQL Execution: The system executes the generated SQL query against the database.

Response Generation: The retrieved data is processed and presented as a natural language response.

Example SQL Query:

SELECT SUM(units) AS total_units_sold

FROM Sales

WHERE product_name = 'Mini Mimi'

AND timestamp >= DATE_SUB(CURDATE(), INTERVAL 7 DAY);

For systems with multiple data tables, an additional step may involve selecting the most relevant table schemas before query execution. This can be achieved through a dedicated table selection model or integrated within the retrieval pipeline.

Evaluating RAG Systems

By combining the strengths of document retrieval with the generative capabilities of large language models, RAG pipelines offer responses that are both contextually rich and dynamically informed. However, the true challenge lies not merely in building such systems, but in rigorously evaluating their performance. In this chapter section, we delve into the performance metrics and evaluation methodologies that ensure RAG systems operate at the highest standards.

• Performance Metrics: Balancing Quality and Efficiency

At the heart of evaluating a RAG system is a suite of performance metrics that assess both the retrieval and generation components. Key among these metrics are:

• Faithfulness: This metric examines how closely the generated response aligns with the information retrieved from the external data sources. A faithful output does not introduce extraneous details or hallucinations; it strictly adheres to the facts contained in the supporting documents. For high-stakes applications, such as legal advice or medical diagnostics, faithfulness is paramount.
• Fluency: Beyond mere factual correctness, the generated output must be linguistically coherent and stylistically appropriate. Fluency assesses grammatical correctness, readability, and overall smoothness of the text, ensuring that the final output is not only accurate but also pleasant and comprehensible for users.
• Recall & Precision: These twin metrics are critical in measuring the effectiveness of the retrieval component. Recall determines the extent to which relevant documents are captured from the corpus, while precision gauges the proportion of those retrieved documents that are indeed pertinent to the query. Together, they provide a balanced view of the retrieval system’s ability to surface the most useful context for the generative model.
• Latency & Efficiency: In real-world applications, response times are as crucial as response quality. Latency measures the time taken to retrieve documents and generate a response, while efficiency reflects the system’s resource utilization. Optimizing these factors ensures that RAG systems can operate seamlessly in production environments without compromising on performance.
• Evaluation Tools: Automated Scoring and Human Insight

To comprehensively evaluate RAG systems, practitioners rely on both automated tools and human expertise. Each approach offers distinct advantages that, when combined, yield a robust evaluation framework.

• Automated Scoring: Tools such as RAGAS, Deepeval, Langsmith, and various OpenAI evaluation frameworks have revolutionized the way we assess RAG outputs. These systems automate the scoring process by analyzing multiple dimensions, including faithfulness, fluency, citation recall, and coherence. Automated scoring not only facilitates scalable and repeatable evaluations but also significantly reduces the manual labor involved in assessing vast quantities of data. By providing quantifiable metrics, these tools allow for fine-grained comparisons between different system configurations and prompt engineering choices.
• Human-in-the-loop Evaluation: Despite the efficiency of automated systems, the nuanced judgment of human experts remains indispensable, especially in high-stakes domains. Expert evaluators are adept at identifying subtle errors, contextual misalignments, and discrepancies that automated tools might overlook. In critical applications, such as financial decision-making, legal research, or healthcare, incorporating human judgment ensures that the RAG system’s outputs meet the highest standards of accuracy and contextual relevance.
• Integrating Evaluation into the RAG Pipeline

A holistic evaluation strategy for RAG systems involves integrating both automated and human assessments into the development cycle. During iterative model refinement, automated scoring tools can rapidly surface potential issues in faithfulness or retrieval accuracy. These insights guide prompt adjustments and hyperparameter tuning. Periodically, human reviewers step in to validate the system’s performance, ensuring that the generated responses not only score well on numerical metrics but also resonate with real-world expectations.

By harmonizing quantitative metrics with qualitative judgment, developers can create RAG systems that are robust, reliable, and ready for deployment in critical applications. The ongoing dialogue between automated evaluations and human expertise forms the backbone of continuous improvement in this exciting field.

Future of RAG

• Scalability Considerations: With increasing data sizes, optimizing indexing and retrieval speed will be crucial to maintaining efficiency in large-scale applications. Advanced indexing techniques and distributed retrieval systems will be key.
• Integration with Multimodal Models: The future of RAG extends beyond text. Combining it with image, video, and structured data retrieval will enable richer, multimodal AI experiences, bridging gaps between textual and visual understanding.
• Next-Generation Retrieval Methods: Advancements in long-context transformers and hybrid retrieval techniques will further refine how AI models synthesize and present knowledge, leading to more efficient and context-aware AI systems.

RAG represents a significant leap forward in AI accuracy and reliability. By integrating retrieval with language generation, it provides dynamic, up-to-date responses that are more factually consistent. This chapter explored its applications, technical implementation, security considerations, and future potential. Implementing robust security practices not only enhances the resilience of AI-driven applications but also ensures compliance with privacy and data protection standards. As AI evolves, RAG will play a foundational role in bridging the gap between static model knowledge and real-world, continuously updated information.