Retrieval Augmented Generation (RAG) for LLMs: A Practical Implementation Guide

It’s the classic LLM demo fail: you ask a frontier model about your company’s Q4 revenue, and it hallucinates a number that belongs in a fantasy novel. Or it confidently explains a deprecated API as if it were still the standard.

Foundation models are frozen in time. They don’t know your private documents, and they don’t know what happened after their training cutoff. Retrieval Augmented Generation (RAG) is the pragmatic engineering answer: you give the LLM an open-book test instead of making it rely on memory.

This guide walks you through the architecture, the code, and the painful lessons learned from shipping RAG in production. We’re skipping the hype and going straight to the implementation.

What Problem Does RAG Actually Solve?

Before we build, let’s define the operational gap. LLMs suffer from two critical flaws in enterprise contexts:

1. Knowledge Cutoff: The model has no access to post-training data.
2. Hallucination: In the absence of ground truth, the model generates plausible-sounding nonsense.

Fine-tuning can help with tone and structure, but it’s a blunt instrument for factual recall—it bakes facts into weights, which are hard to update and can cause catastrophic forgetting.

RAG flips the script. Instead of asking the model to recall facts, we ask it to reason over provided facts.

The Core RAG Architecture (with Diagram)

At a high level, RAG is a two-phase process: ingestion and querying.

During ingestion, you break down source documents, embed them into vectors, and store them in a vector database. During querying, you embed the user’s question, find the nearest neighbor vectors, and stuff those relevant chunks into the LLM’s prompt.

The magic isn’t in any single component—it’s in the pipeline’s ability to inject ground truth at runtime. If a document changes, you just re-index that snippet. The model weights stay untouched.

Step-by-Step Implementation: From PDF to Answer

Let’s build a minimal RAG pipeline in Python. We’ll use LangChain for orchestration, OpenAI for embeddings and generation, and ChromaDB as our local vector store.

Dependencies:

pip install langchain openai chromadb pypdf tiktoken

1. Load and Chunk Documents

from langchain.document_loaders import PyPDFLoader
from langchain.text_splitter import RecursiveCharacterTextSplitter

loader = PyPDFLoader("q4_report.pdf")
documents = loader.load()

text_splitter = RecursiveCharacterTextSplitter(
    chunk_size=1000,
    chunk_overlap=200,
    length_function=len,
)
chunks = text_splitter.split_documents(documents)

2. Embed and Store

from langchain.embeddings import OpenAIEmbeddings
from langchain.vectorstores import Chroma

embeddings = OpenAIEmbeddings()
vectorstore = Chroma.from_documents(
    documents=chunks,
    embedding=embeddings,
    persist_directory="./chroma_db"
)

3. Retrieve and Generate

from langchain.chains import RetrievalQA
from langchain.chat_models import ChatOpenAI

llm = ChatOpenAI(model_name="gpt-4-turbo", temperature=0)
qa_chain = RetrievalQA.from_chain_type(
    llm=llm,
    chain_type="stuff",
    retriever=vectorstore.as_retriever(search_kwargs={"k": 4})
)

response = qa_chain.run("What was our revenue growth in Q4?")
print(response)

This is the “hello world” of RAG. It works for demos. Production is harder.

Chunking Strategies: The Silent Make-or-Break Factor

The chunk_size and chunk_overlap numbers above are the most critical hyperparameters in your pipeline. If your chunks are too small, you lose semantic context. If they’re too large, you drown the retriever in noise.

There is no magic number, but here are empirical heuristics for different content types:

The Overlap Trap: Overlap prevents a single sentence from being cut in half, but too much overlap creates a “needle in a haystack” problem where every retrieved chunk looks identical. If you see low diversity in your retrieved context, reduce the overlap.

For a deep dive on structuring data for local inference, check out our guide on Extracting Invoices to Structured JSON with Ollama, which covers chunking strategies for messy financial documents.

Retrieval: Dense vs. Sparse vs. Hybrid Search

Not all search is created equal. The naive approach uses dense vector search (cosine similarity on embeddings). It works well for semantic meaning but fails catastrophically on exact keyword matches (e.g., error codes, serial numbers).

• Dense (Vector): Embeddings. Great for “What is the policy on refunds?”
• Sparse (BM25/TF-IDF): Keyword matching. Great for “Error code 0xE0434352”.
• Hybrid: Combines both using Reciprocal Rank Fusion (RRF).

For production, implement hybrid search. Use a vector DB that supports it natively (Weaviate, Pinecone, Elasticsearch) or combine results manually:

def hybrid_score(dense_results, sparse_results, alpha=0.7):
    # alpha controls weight of dense vs sparse
    combined = {}
    for rank, doc in enumerate(dense_results):
        combined[doc.id] = alpha * (1 / (rank + 1))
    for rank, doc in enumerate(sparse_results):
        if doc.id in combined:
            combined[doc.id] += (1 - alpha) * (1 / (rank + 1))
        else:
            combined[doc.id] = (1 - alpha) * (1 / (rank + 1))
    return sorted(combined.items(), key=lambda x: x[1], reverse=True)

Prompt Engineering the Augmented Context

Once you retrieve the top-k chunks, you need to stuff them into the LLM’s context window. The naive “stuff” chain dumps everything in. For longer documents, use Map-Reduce or Refine.

But the real art is in the system prompt. You must constrain the model to only use the provided context and explicitly signal when it doesn’t know.

High-Signal RAG Prompt Template:

You are a precise assistant. Answer the question based ONLY on the context below.
If the context does not contain the answer, say "I don't have that information in the provided documents."
Do not use prior knowledge.

Context:
{context}

Question: {question}

Answer:

Adding a citation requirement forces the model to ground its answers:

Provide the answer and cite the source chunk IDs in square brackets.

Evaluating Your RAG Pipeline: Beyond Vibes

You cannot improve what you don’t measure. RAG evaluation has two dimensions:

1. Retrieval Quality: Did we get the right chunks?
2. Generation Quality: Did the LLM use them correctly?

Start with a golden dataset of 50-100 question-answer pairs. Run your pipeline, log the retrieved chunks, and manually score them. Automate later.

Production Considerations: Latency, Cost, and Guardrails

Shipping RAG to production isn't just about accuracy; it's about keeping the system online and your costs predictable.

• Latency Budget: Embedding + Vector Search + LLM Inference. If you’re on a 2-second P95 budget, you need sub-100ms retrieval. Use approximate nearest neighbor (ANN) indexes (HNSW) and avoid re-ranking unless you have headroom.
• Caching: Cache embeddings for frequent queries. Cache LLM responses for identical queries. A simple Redis cache can cut costs by 40%.
• Guardrails: The retrieved context might contain PII or toxic content. Implement a lightweight filter between retrieval and generation. For a real-world example of shipping an LLM feature with guardrails in a restricted environment, read our case study on Shipping an LLM Feature at a Bank in 5 Days.
• Streaming: Use server-sent events (SSE) to stream tokens. It masks latency and makes the UX feel responsive.

If you’re building internal tools to automate engineering workflows, you might also find our guide on Building an On-Call Incident Summarizer useful—it applies these exact RAG patterns to log analysis.

FAQ: RAG for LLMs

Is ChatGPT a RAG LLM?

ChatGPT’s default mode is not RAG; it relies on its training data. However, when you use the browsing plugin or the “Search” feature in ChatGPT Plus, it effectively becomes a RAG system—retrieving live web pages, chunking them, and augmenting the prompt before generating a response.

How does RAG work with LLMs?

RAG intercepts a user query before it reaches the LLM. It vectorizes the query, searches a knowledge base for semantically similar content, and injects that content into the prompt as “ground truth.” The LLM then acts as a reasoning engine over that provided text rather than relying on its internal weights.

Is RAG Retrieval-Augmented Generation?

Yes. The term was coined in the 2020 paper “Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks” by Lewis et al. at Facebook AI Research. It describes the specific architecture of coupling a retriever (usually DPR) with a generator (like BART or an LLM).

How to implement RAG in LLM?

The simplest path: use LangChain or LlamaIndex to connect a document loader, a text splitter, an embedding model (like text-embedding-3-small), and a vector database. For a full-stack approach, check out the step-by-step implementation section above, or explore our GitHub PR Review Bot guide which uses a similar retrieval pattern for code reviews.

What is the biggest bottleneck in RAG?

Chunking strategy and retrieval quality. Most failures aren’t model failures; they’re “garbage in, garbage out” failures where the retriever fetches irrelevant text. Focus on your evaluation pipeline before fine-tuning the LLM.

Can I use open-source models for RAG?

Absolutely. Pair a local embedding model (like BAAI/bge-small-en) with an open-source LLM (like Llama 3) served via Ollama. The architecture is model-agnostic. For an example of shipping open-source models locally, see our job application autofill extension build.