Stop AI Hallucinations: Build a RAG Chatbot That Reads Before It Answers

Why Standard AI Chatbots Lie

Without access to your documents, AI models answer from training data memory, guessing when uncertain. This creates confident false answers called hallucinations—the model invents information rather than admitting it doesn't know.

Same model, same question—RAG forces the AI to read the source first

One-Command Installation

A single terminal command installs three core tools: LangChain (orchestration), Chroma (vector database), and LangChain Google GenAI (Gemini integration). Grab a free Gemini API key and paste it into your code to authenticate.

Three-step setup before coding begins

Load Document into Python

Use TextLoader to read your document file (e.g., leave_policy.txt) into Python. For PDFs, swap TextLoader for PyPDFLoader. This pulls the entire file content into a variable for processing.

Split into Searchable Chunks

Documents are too large to search efficiently whole. Use RecursiveCharacterTextSplitter to break the document into small pieces. Set chunk_size to 300 (a few sentences) and chunk_overlap to 50 so sentences cut at boundaries aren't lost.

300

Characters per chunk (optimal for sentence-level meaning)

Chunk size balances context and searchability

What Are Embeddings

An embedding is a list of numbers that captures the semantic meaning of a chunk. Chunks about vacation days get similar number patterns; chunks about sick leave get different ones. This numeric representation allows meaning-based search.

Generate Embeddings with Gemini

Use Gemini's embedding model (e.g., 'models/embedding-001') to convert each chunk into a vector. The same Gemini API key handles both embeddings and later answer generation. If you encounter a model-not-found error, Google has renamed the model—grab the current name from AI Studio.

Build Your Knowledge Base with Chroma

Use Chroma's from_documents method to store all chunks and their embeddings in a vector database. A vector database is optimized to hold number lists and search them by similarity at scale. Once stored, your knowledge base is ready to retrieve.

Set Up the Retriever

Create a retriever that searches the vector database when a question arrives. Set k=3 to return the top 3 most relevant chunks by meaning. Raise k if answers are missed; lower it if irrelevant chunks appear.

3

Top chunks retrieved per question (k parameter)

Balance between context richness and noise

Configure the Language Model

Initialize ChatGoogleGenerativeAI with temperature=0 to disable creativity and enforce factual answers. Use Gemini 2.5 Flash for speed and free-tier compatibility. Temperature is a creativity dial: 0 means strictly factual, which is essential for RAG.

0

Temperature setting (factual mode, no embellishment)

Zero creativity ensures grounded, document-based answers

The RAG Pipeline: Retrieve Then Generate

When a question arrives, retrieve the top k chunks from the vector database, paste them into the prompt, and instruct the model to answer using only those chunks. The model reads the exact lines from your file instead of relying on training memory. This is the core trick that eliminates hallucinations.

RAG workflow: Retrieve, then Generate

Add Transparency with Source Citations

Print the chunk used to generate each answer alongside the response. This gives every answer a receipt—users can verify the source and check accuracy themselves.

The Hallucination Root Cause

The broken bot at the start was taking a closed-book exam, answering from training data memory. When uncertain, it guessed—that guess is a hallucination. RAG converts it to an open-book exam: retrieve the right page first, then answer from it.

Same model, different approach—retrieval eliminates hallucinations

What RAG Stands For

RAG is Retrieval Augmented Generation. The model retrieves relevant context first, then generates answers augmented by that context. It reads before it speaks.

Three Cases Where RAG Doesn't Apply

Skip RAG if data is tiny (paste directly in prompt), if you need live numbers like prices or weather (use an API instead), or if the task is creative like writing (RAG is for facts, not imagination).

RAG is for fact-grounded tasks, not all AI use cases