Build a Local RAG Chatbot Over Your PDFs with Ollama, LlamaIndex, and Qdrant Free Tier

What We're Building

A fully private, local-first retrieval-augmented generation chatbot that sits on top of your PDF library. You drop PDFs into a folder, run the ingestion script once, then ask natural-language questions against the entire corpus. The LLM never sees your documents during training—it only receives relevant chunks at inference time, served by a vector search engine.

Feature list:

• Ingestion pipeline that chunks PDFs and generates embeddings with Ollama
• Qdrant cloud vector database storing embeddings (free tier: 1GB, no credit card for sandbox)
• Local LlamaIndex query engine that retrieves top-k chunks and synthesizes an answer
• Interactive terminal chat loop with source attribution
• Zero egress costs—all inference stays on your machine

Architecture Overview

The flow is straightforward: documents get chunked, each chunk is embedded by an Ollama model, and vectors land in Qdrant. At query time, the user question is embedded against the same model, Qdrant returns the most semantically similar chunks, and those chunks are stuffed into the prompt for the Ollama LLM to generate a grounded answer.

Prerequisites

All tools are free-tier or open-source. No API keys that cost money.

Hardware note: You need enough RAM to run a 7B-parameter model comfortably. 16GB RAM works; 8GB is tight but doable with a 3B model. An Apple Silicon Mac or any modern x86 CPU with AVX2 will work. GPU acceleration is optional—Ollama handles CPU inference gracefully.

Step 1: Spin Up Ollama and Pull a Model

Install Ollama from the link above. Once installed, pull a model that supports both text generation and embeddings. We'll use nomic-embed-text for embeddings and llama3.2:3b for chat (small, fast, free).

ollama pull nomic-embed-text
ollama pull llama3.2:3b

Verify they're available:

ollama list

Ollama runs as a background service on localhost:11434. Confirm it's alive:

curl http://localhost:11434/api/tags

You should see both models in the JSON response.

Step 2: Provision Qdrant Cloud Free Tier

Head to cloud.qdrant.io and sign up with GitHub or email. Create a new cluster:

• Choose the Free plan (1GB storage, no payment required)
• Pick a region close to you
• Name it something like pdf-rag-cluster

Once provisioned, grab your Cluster URL and API Key from the dashboard. The URL looks like https://xyz-example.qdrant.io. The API key is a long string you'll paste into your environment.

Security note: The free tier is publicly routed but authenticated via API key. For truly sensitive documents, consider running Qdrant locally via Docker instead. The code structure stays identical—just swap the URL to http://localhost:6333 and drop the API key.

Step 3: Initialize the Python Environment

Create a project directory and a virtual environment:

mkdir pdf-rag-chatbot
cd pdf-rag-chatbot
python -m venv venv
source venv/bin/activate  # Windows: venv\Scripts\activate

Install dependencies:

pip install llama-index llama-index-vector-stores-qdrant llama-index-embeddings-ollama llama-index-llms-ollama pypdf python-dotenv

Create a .env file for Qdrant credentials:

QDRANT_URL=https://xyz-example.qdrant.io
QDRANT_API_KEY=your-api-key-here

Create a subfolder for your PDFs:

mkdir documents

Drop any PDFs you want to index into documents/. Research papers, manuals, contracts—anything text-based works. Scanned image PDFs need OCR (see Extensions).

Step 4: Ingest PDFs, Chunk, and Embed

Create ingest.py:

import os
from dotenv import load_dotenv
from llama_index.core import SimpleDirectoryReader, Settings
from llama_index.core.node_parser import SentenceSplitter
from llama_index.embeddings.ollama import OllamaEmbedding
from llama_index.vector_stores.qdrant import QdrantVectorStore
from qdrant_client import QdrantClient

load_dotenv()

# Configure global settings
Settings.embed_model = OllamaEmbedding(model_name="nomic-embed-text", base_url="http://localhost:11434")
Settings.chunk_size = 512
Settings.chunk_overlap = 50

# Load PDFs
documents = SimpleDirectoryReader("documents", required_exts=[".pdf"]).load_data()
print(f"Loaded {len(documents)} documents.")

# Chunk
parser = SentenceSplitter(chunk_size=512, chunk_overlap=50)
nodes = parser.get_nodes_from_documents(documents)
print(f"Created {len(nodes)} chunks.")

# Connect to Qdrant
client = QdrantClient(url=os.getenv("QDRANT_URL"), api_key=os.getenv("QDRANT_API_KEY"))
vector_store = QdrantVectorStore(client=client, collection_name="pdf_rag")

# Embed and store
vector_store.add(nodes)
print("Ingestion complete. Vectors stored in Qdrant.")

Run it:

python ingest.py

What's happening: SimpleDirectoryReader extracts text from each PDF. SentenceSplitter breaks text into 512-token chunks with 50-token overlap to preserve context across boundaries. Each chunk is sent to Ollama's nomic-embed-text model (runs locally), which returns a 768-dimensional vector. Those vectors are upserted into Qdrant under the collection pdf_rag.

Re-ingestion safety: If you add new PDFs, re-run the script. By default, LlamaIndex's add() method upserts nodes by ID, so duplicate chunks are overwritten rather than duplicated. If you want a full reset, delete the collection in the Qdrant dashboard first.

Step 5: Build the Retrieval Engine

Create query.py with the retrieval logic:

import os
from dotenv import load_dotenv
from llama_index.core import Settings, VectorStoreIndex
from llama_index.embeddings.ollama import OllamaEmbedding
from llama_index.llms.ollama import Ollama
from llama_index.vector_stores.qdrant import QdrantVectorStore
from qdrant_client import QdrantClient

load_dotenv()

# Configure
Settings.embed_model = OllamaEmbedding(model_name="nomic-embed-text", base_url="http://localhost:11434")
Settings.llm = Ollama(model="llama3.2:3b", base_url="http://localhost:11434", request_timeout=120.0)

# Connect to existing Qdrant collection
client = QdrantClient(url=os.getenv("QDRANT_URL"), api_key=os.getenv("QDRANT_API_KEY"))
vector_store = QdrantVectorStore(client=client, collection_name="pdf_rag")

# Build index from existing vectors (no re-embedding)
index = VectorStoreIndex.from_vector_store(vector_store)

# Create query engine with top-3 retrieval
query_engine = index.as_query_engine(similarity_top_k=3)

# Interactive loop
print("PDF RAG Chatbot ready. Type 'exit' to quit.\n")
while True:
    query = input("You: ")
    if query.lower() in ("exit", "quit"):
        break
    response = query_engine.query(query)
    print(f"\nBot: {response}\n")
    if hasattr(response, 'source_nodes'):
        for i, node in enumerate(response.source_nodes):
            print(f"Source {i+1}: {node.metadata.get('file_name', 'unknown')} (score: {node.score:.3f})")
        print()

Key detail: VectorStoreIndex.from_vector_store() connects to an already-populated Qdrant collection. It does not re-embed anything—it just reads the existing vectors and wraps them in a LlamaIndex index object. This is what makes the query script fast to initialize.

Step 6: Assemble the Chat Loop

The query engine already handles the full RAG chain: embed query → retrieve chunks → prompt LLM → return answer. The as_query_engine(similarity_top_k=3) method creates a default prompt template that includes the retrieved chunks and instructs the model to answer based on them.

If you want more control over the prompt, you can swap in a custom template:

from llama_index.core.prompts import PromptTemplate

custom_template = PromptTemplate(
    "You are a precise research assistant. Answer the question using ONLY the context below. "
    "If the context doesn't contain the answer, say 'I don't have enough information.'\n\n"
    "Context:\n{context_str}\n\nQuestion: {query_str}\nAnswer:"
)
query_engine.update_prompts({"response_synthesizer:text_qa_template": custom_template})

This tightens up the model's behavior and reduces hallucination outside the document scope.

Running the Chatbot

python query.py

You'll see:

PDF RAG Chatbot ready. Type 'exit' to quit.

You: What is the main finding of the 2024 paper on attention mechanisms?

Bot: The paper finds that multi-head attention can be pruned by 40% without loss in perplexity...
Source 1: attention_survey_2024.pdf (score: 0.892)
Source 2: transformer_impl.pdf (score: 0.845)

Latency depends on your hardware. On an M1 MacBook Pro, llama3.2:3b generates roughly 30-40 tokens per second. Embedding is faster—hundreds of chunks per second.

Sensible Extensions

Add a web UI. Wrap the query engine in a Streamlit app (free, open-source) for a ChatGPT-like interface. Streamlit's chat_message and st.chat_input components make this a 30-minute add-on.

OCR for scanned PDFs. If your PDFs are image-based, pipe them through Tesseract OCR (free, open-source) before ingestion. LlamaIndex doesn't natively handle image PDFs, so preprocess with pytesseract and save extracted text as .txt files alongside the PDFs.

Hybrid search. Qdrant supports sparse vectors (BM25) alongside dense embeddings. Combine keyword and semantic search for better retrieval on technical documents with lots of jargon. LlamaIndex has a QdrantHybridRetriever for this.

Multi-user with session history. Store conversation history per user and use a ChatEngine instead of a QueryEngine. LlamaIndex's ContextChatEngine maintains a rolling window of previous Q&A pairs, enabling follow-up questions like "summarize that in bullet points."

Swap to a larger model. When you need more reasoning power, pull llama3.1:8b or mistral:7b and change the model name in Settings.llm. No code changes needed.

If you're interested in building more AI-powered tools that run on free infrastructure, check out Build a Smart Clipboard that Translates and Summarizes Text with Groq and Piper TTS or Build a Screenshot-to-React Agent with Google Gemini Flash and Free Hosting. For a deeper dive into production-grade retrieval patterns, Beating GPT-5.6 Sol on Retrieval with 100x Cheaper Open Models: The Castform Stack walks through the engineering decisions that make open-source RAG competitive.

Common Pitfalls and How to Avoid Them

FAQ

Q: Does this send my PDFs to any cloud service? A: The PDF text is chunked and embedded locally. Only the embedding vectors (arrays of floats) are sent to Qdrant Cloud. The original text never leaves your machine unless you choose to use a cloud LLM. With Ollama, everything stays local.

Q: Can I use a different embedding model? A: Yes. Any model Ollama supports works. mxbai-embed-large gives higher quality embeddings at the cost of more RAM. Swap the model name in OllamaEmbedding(model_name="...") and re-ingest.

Q: What if I want to run Qdrant entirely locally? A: Spin up Qdrant via Docker: docker run -p 6333:6333 qdrant/qdrant. Then set QDRANT_URL=http://localhost:6333 and omit the API key. The code is identical.

Q: How do I update the index when I add new PDFs? A: Re-run ingest.py. It upserts by node ID, so existing chunks stay untouched and new ones are added. No need to rebuild the entire index.

Q: Can I query across multiple collections? A: LlamaIndex supports this natively. Create separate collections for different document types, then use a RouterQueryEngine to route questions to the right collection based on keyword matching or LLM-based classification.

Q: The answers feel generic. How do I improve retrieval quality? A: Tune chunk_size and chunk_overlap for your document type. Dense technical docs benefit from smaller chunks (256). Narrative content works better with larger chunks (1024). Also experiment with similarity_top_k—more chunks give the LLM more context but can dilute relevance.

Q: Is this production-ready? A: For personal use and internal tools, absolutely. For customer-facing production, you'd want to add error handling, logging, rate limiting, and likely upgrade to a paid Qdrant plan for higher availability. The architecture pattern scales directly. For insight on when and how to hand off a prototype like this to core engineering, see Scaling Yourself: When an FDE Hands Off to Core Engineering for Productionization.