Corrective RAG: Fixing Traditional RAG's Hallucination Problem
Corrective RAG (CRAG) solves traditional RAG's blind trust in retrieved documents by adding a retrieval evaluator that classifies documents as relevant, irrelevant, or ambiguous—then routes to refinement, web search, or hybrid approaches accordingly. This prevents hallucinations when documents don't match queries.
The Problem with Traditional RAG
Blind Trust in Retrieved Documents
Traditional RAG blindly trusts whatever documents the retriever returns and sends them directly to the LLM. If the retrieved documents don't actually answer the query, the LLM is forced to generate wrong answers from irrelevant context, leading to hallucinations.
Real-World Example: Transformer Query Failure
When asked 'What is a Transformer in Deep Learning?' on a vector database containing only classic ML books (Hands-On Machine Learning, Deep Learning, Pattern Recognition), the retriever returned documents about MLPs and CNNs instead. The LLM still generated a plausible-sounding answer by relying on its parametric knowledge rather than the retrieved context, demonstrating the hallucination risk.
Business Risk of Hallucinations
In enterprise scenarios, hallucinations can be dangerous. For example, if an employee asks about leave policy but the vector database has no relevant documents, the system will confidently provide incorrect information that the employee acts upon, with serious consequences.
Traditional RAG Workflow Overview
Three-Step RAG Process
RAG consists of retrieval (converting user query to vector and finding similar documents in vector database), augmentation (combining query with retrieved documents), and generation (LLM produces answer from augmented context).
Corrective RAG Architecture
Core Innovation: Retrieval Evaluator
CRAG adds a retrieval evaluator model that assesses whether retrieved documents are actually relevant to the query before sending them to the LLM. This evaluator scores each document and makes routing decisions based on relevance thresholds.
Three-Case Classification System
The retrieval evaluator classifies retrieved documents into three cases: (1) Correct—at least one document scores above upper threshold (7), proceed with normal RAG; (2) Incorrect—all documents score below lower threshold (3), trigger web search; (3) Ambiguous—mixed scores between thresholds, use both retrieved and web-searched documents.
Routing Logic Based on Evaluation
If correct: refine documents and generate answer. If incorrect: rewrite query, search web, refine web results, then generate. If ambiguous: combine retrieved documents (scoring >3) with web search results, refine both, then generate.
Knowledge Refinement Feature
Three-Step Refinement Process
Knowledge refinement cleans retrieved documents by: (1) Decomposition—breaking documents into sentence-level strips; (2) Filtration—using an LLM or T5 model to score each strip's relevance to the query; (3) Recombination—merging only relevant strips back into a refined context.
Why Refinement Matters
During text chunking, documents are split at fixed character boundaries (e.g., 900 chars) without semantic awareness. A single chunk may contain both relevant and irrelevant content. Refinement removes the irrelevant parts, improving generation quality.
Refinement Example
A document chunk about gradient descent may also contain unrelated sentences about neural networks and CNNs from adjacent paragraphs. Refinement decomposes it into four sentences, filters to keep only the two about gradient descent, and recombines them into a clean context.
Retrieval Evaluation Scoring
Document Scoring Mechanism
The evaluator sends each retrieved document and the query to an LLM (or fine-tuned T5), which returns a relevance score between 0 and 1, where 1 means the document alone fully answers the query and 0 means it's irrelevant. The paper uses thresholds of 3 (lower) and 7 (upper) on a 0-10 scale.
Good Documents Filtering
Only documents with evaluation scores greater than the lower threshold (3) are used for generation. Documents scoring below 3 are discarded. This prevents weak or marginally relevant documents from contaminating the final answer.
Threshold-Based Decision Rules
If any document scores >7 (upper threshold): verdict is Correct. If all documents score <3 (lower threshold): verdict is Incorrect. If scores fall between thresholds or mix above and below: verdict is Ambiguous.
Web Search Integration
Fallback to External Knowledge
When retrieval evaluation returns Incorrect or Ambiguous, CRAG triggers a web search via Tavily API to fetch external documents. These web results are then refined using the same knowledge refinement process before generation.
Query Rewriting for Search Optimization
Before web search, the original user query is rewritten into a search-engine-optimized query. For example, 'LLM and Recent Developments' becomes 'LLM Recent Developments Last 30 Days'. This improves search result quality by adding specificity and temporal constraints.
Real-World Example: AI News Query
When asked 'What are the top AI news from last month?' on a database of classic ML books, retrieval evaluation returns Incorrect (no relevant documents). CRAG rewrites the query, searches the web, retrieves current AI news, refines it, and generates an accurate answer instead of hallucinating.
Implementation Architecture
Iterative Development Approach
The full CRAG system is built incrementally: Iteration 1 adds knowledge refinement to traditional RAG; Iteration 2 adds retrieval evaluation; Iteration 3 adds web search; Iteration 4 adds query rewriting; Iteration 5 handles the ambiguous case by combining retrieved and web documents.
LangGraph Implementation
CRAG is implemented using LangGraph, a framework for building stateful multi-node workflows. The graph contains nodes for retrieval, evaluation, refinement, web search, query rewriting, and generation, with conditional routing based on evaluation verdicts.
State Management
The LangGraph state tracks: question, retrieved documents, strips (sentence-level chunks), kept strips (filtered strips), refined context, good documents (scoring >3), web documents, web query, verdict, and final answer. This enables seamless data flow between nodes.
Handling the Ambiguous Case
Hybrid Approach for Mixed Signals
When evaluation returns Ambiguous (some documents score >3 but none >7), CRAG combines both retrieved documents and web search results. It refines both together, creating a unified context that leverages both internal and external knowledge.
Example: Batch Normalization vs Layer Normalization
Query on a database where batch normalization is covered but layer normalization is not returns Ambiguous (mixed relevance scores). CRAG retrieves partial documents, searches the web for layer normalization info, combines both, refines, and generates a complete answer.
Simplified Two-Case Routing
Rather than three separate paths, the implementation uses state intelligence to merge Incorrect and Ambiguous into a single web-search path. The refine node intelligently uses either good documents alone (Correct), web documents alone (Incorrect), or both (Ambiguous).
Key Differences from Paper Implementation
T5 vs LLM Trade-off
The original CRAG paper uses a fine-tuned T5-Large transformer (770M parameters) for filtration and evaluation tasks because it's lightweight, cheaper, and performs better on these specific tasks. The implementation uses OpenAI's LLM instead due to lack of access to the fine-tuned T5 model.
Threshold Selection
The paper does not explicitly specify threshold values. The implementation uses lower threshold of 3 and upper threshold of 7 (on a 0-10 scale), which can be tuned based on use case requirements.
Practical Simplifications
The implementation consolidates the three-case routing into two main paths (Correct vs Web Search) using state management, making the code simpler while preserving the architectural logic.
Practical Outcomes
Correct Case: Bias-Variance Trade-off Query
Query 'What is bias-variance trade-off?' on ML books returns Correct verdict (retrieved documents score >7). System refines documents, generates accurate answer without web search, maintaining low latency and cost.
Incorrect Case: Out-of-Domain Query
Query 'What are top AI news from last month?' on classic ML books returns Incorrect verdict (all documents score <3). System rewrites query, searches web, retrieves current news, and generates timely answer instead of hallucinating.
Ambiguous Case: Partially Covered Topic
Query 'Batch Normalization vs Layer Normalization' where only batch norm is in books returns Ambiguous verdict (mixed scores). System combines retrieved batch norm docs with web-searched layer norm info, refines both, and generates comprehensive answer.
Notable quotes
If the documents are not relevant then whatever answer will be generated will also be wrong. — Nitesh
Corrective RAG does not assume that whatever documents are retrieved are correct. — Nitesh
Even if our documents are not there or not sufficient, we don't want to send the user back empty handed. — Nitesh
Action items
- Implement traditional RAG baseline with retrieval, augmentation, and generation steps using LangChain
- Add knowledge refinement node: decompose documents into sentences, filter by relevance using LLM, recombine into refined context
- Build retrieval evaluator using LLM to score each document 0-10 and classify as Correct (>7), Incorrect (<3), or Ambiguous (3-7)
- Integrate Tavily web search API for Incorrect and Ambiguous cases, with query rewriting before search
- Implement LangGraph state management to route between retrieval, evaluation, refinement, web search, and generation nodes
- Test on out-of-domain queries to verify web search fallback prevents hallucinations
- Tune lower and upper thresholds (default 3 and 7) based on your domain and quality requirements
- Read the original CRAG paper (2024) for detailed architecture and fine-tuning approaches with T5-Large