Build a RAG Chatbot Over Your Own PDFs and Notes Using Qdrant Free Tier

What We’re Building

A local retrieval-augmented generation (RAG) pipeline that sucks in your messy folder of PDFs and markdown notes, chunks them intelligently, stuffs the embeddings into a free Qdrant vector database, and lets you interrogate your own knowledge base with natural language. Every answer comes with cited sources so you know exactly which document and chunk the model pulled from.

Feature list:

• Ingests PDFs and .md files from a local directory
• Chunks documents with configurable size and overlap
• Generates embeddings via a free, fast model
• Stores vectors in Qdrant Cloud’s free tier (1GB, 1 cluster)
• Queries using Groq’s free tier running Mixtral 8x7B at absurd token speeds
• Returns answers with inline citations pointing back to source files
• Runs entirely from a single Python script—no orchestration layer required

Architecture Overview

The flow is dead simple: documents land on disk, LlamaIndex reads and chunks them, an embedding model converts each chunk to a vector, Qdrant stores those vectors, and at query time we pull the most relevant chunks and feed them to Groq’s Mixtral as context.

No GPU needed. No Docker if you don’t want it. The only external services are Qdrant Cloud and Groq Cloud, both with generous free tiers that won’t bill you unless you explicitly upgrade.

Prerequisites

Everything here is free-tier or open-source. You’ll need:

• Python 3.10+ – python.org/downloads
• Qdrant Cloud account – qdrant.tech → sign up, create a free cluster (1GB storage, no credit card). Copy your cluster URL and API key.
• Groq Cloud API key – console.groq.com → sign up, grab a free API key. The free tier gives you enough tokens per minute to run dozens of queries.
• A folder of PDFs and/or .md files – Throw in whatever: research papers, meeting notes, documentation, personal wiki exports.

That’s it. No vector database to host locally, no GPU to spin up, no credit card to hand over.

Step 1: Project Setup and Dependencies

Create a project directory and a virtual environment. Then install the core packages:

mkdir rag-qdrant-groq && cd rag-qdrant-groq
python -m venv venv
source venv/bin/activate  # Windows: venv\Scripts\activate

pip install llama-index llama-index-vector-stores-qdrant llama-index-llms-groq llama-index-embeddings-fastembed qdrant-client pypdf

What each package does:

• llama-index – Core orchestration: loading, chunking, indexing, querying.
• llama-index-vector-stores-qdrant – Qdrant integration for LlamaIndex.
• llama-index-llms-groq – Groq LLM integration.
• llama-index-embeddings-fastembed – FastEmbed models (all-MiniLM-L6-v2 by default). Runs locally, no API cost, surprisingly good.
• qdrant-client – Direct Qdrant client (used under the hood).
• pypdf – PDF parsing.

Set your API keys as environment variables. Create a .env file (add it to .gitignore immediately):

# .env
QDRANT_URL=https://your-cluster-url.qdrant.io
QDRANT_API_KEY=your-qdrant-api-key
GROQ_API_KEY=gsk_your_groq_api_key

Load them in your script with python-dotenv (install it: pip install python-dotenv).

Step 2: Ingesting PDFs and Markdown Notes

Create ingest.py. We’ll use LlamaIndex’s SimpleDirectoryReader which handles PDFs, markdown, and plain text out of the box.

# ingest.py
import os
from dotenv import load_dotenv
from llama_index.core import SimpleDirectoryReader, Settings
from llama_index.embeddings.fastembed import FastEmbedEmbedding
from llama_index.vector_stores.qdrant import QdrantVectorStore
from qdrant_client import QdrantClient

load_dotenv()

# Set the global embedding model
Settings.embed_model = FastEmbedEmbedding(model_name="BAAI/bge-small-en-v1.5")

# Load documents from a directory
DOCS_DIR = "./my_docs"  # Point this at your folder
documents = SimpleDirectoryReader(DOCS_DIR, recursive=True).load_data()
print(f"Loaded {len(documents)} documents")

Chunking strategy: BGE-small-en-v1.5 has a max context of 512 tokens, so chunk accordingly. LlamaIndex defaults to 1024-token chunks with 20-token overlap. That works, but for dense technical docs, smaller chunks with more overlap often yield better retrieval. Let’s set it explicitly:

from llama_index.core.node_parser import SentenceSplitter

Settings.text_splitter = SentenceSplitter(chunk_size=512, chunk_overlap=50)

Step 3: Indexing Documents into Qdrant

Now connect to Qdrant Cloud and push the vectors. The free tier gives you one cluster. We’ll create a collection called my_knowledge_base.

# Initialize Qdrant client
client = QdrantClient(
    url=os.getenv("QDRANT_URL"),
    api_key=os.getenv("QDRANT_API_KEY"),
)

# Create vector store
vector_store = QdrantVectorStore(
    client=client,
    collection_name="my_knowledge_base",
)

# Create index from documents
from llama_index.core import VectorStoreIndex, StorageContext

storage_context = StorageContext.from_defaults(vector_store=vector_store)
index = VectorStoreIndex.from_documents(
    documents,
    storage_context=storage_context,
    show_progress=True,
)

print(f"Indexed {len(documents)} documents into Qdrant")

Run python ingest.py. First run downloads the embedding model (~130MB), then chunks and embeds everything. For a few hundred pages, this takes under a minute on a laptop.

Verify in Qdrant Cloud dashboard: Go to your cluster, open the Collections tab, and you’ll see my_knowledge_base with the correct vector count.

Step 4: Building the Query Engine with Groq

Create query.py. We’ll load the existing index from Qdrant (no re-indexing) and wire up Groq’s Mixtral 8x7B as the response synthesizer.

# query.py
import os
from dotenv import load_dotenv
from llama_index.core import Settings, VectorStoreIndex, StorageContext
from llama_index.embeddings.fastembed import FastEmbedEmbedding
from llama_index.llms.groq import Groq
from llama_index.vector_stores.qdrant import QdrantVectorStore
from qdrant_client import QdrantClient

load_dotenv()

# Re-use the same embedding model used during ingestion
Settings.embed_model = FastEmbedEmbedding(model_name="BAAI/bge-small-en-v1.5")

# Groq LLM - Mixtral 8x7B is fast and free-tier friendly
Settings.llm = Groq(
    model="mixtral-8x7b-32768",
    api_key=os.getenv("GROQ_API_KEY"),
    temperature=0.1,  # Low temp for factual answers
)

# Connect to existing Qdrant collection
client = QdrantClient(
    url=os.getenv("QDRANT_URL"),
    api_key=os.getenv("QDRANT_API_KEY"),
)
vector_store = QdrantVectorStore(
    client=client,
    collection_name="my_knowledge_base",
)

# Load index from vector store (no re-indexing)
storage_context = StorageContext.from_defaults(vector_store=vector_store)
index = VectorStoreIndex.from_vector_store(
    vector_store,
    storage_context=storage_context,
)

# Build query engine with citation support
query_engine = index.as_query_engine(
    similarity_top_k=4,  # Retrieve top 4 chunks
    response_mode="compact",  # Compact mode fits context window efficiently
    verbose=True,
)

# Interactive query loop
print("RAG Chatbot ready. Type 'exit' to quit.\n")
while True:
    query = input("Ask a question: ")
    if query.lower() == "exit":
        break
    response = query_engine.query(query)
    print(f"\nAnswer: {response.response}\n")
    print("Sources:")
    for node in response.source_nodes:
        file_name = node.metadata.get("file_name", "unknown")
        page = node.metadata.get("page_label", "N/A")
        print(f"  - {file_name} (page {page}), score: {node.score:.3f}")
    print("\n" + "-" * 50 + "\n")

What’s happening under the hood:

1. Your question is embedded using the same BGE-small model.
2. Qdrant runs an approximate nearest-neighbor search, returning the 4 most semantically similar chunks.
3. Those chunks are stuffed into a prompt template along with your question.
4. Groq’s Mixtral generates a response grounded in those chunks.
5. LlamaIndex extracts source metadata and returns it alongside the answer.

Step 5: Running the Chatbot

python query.py

You’ll see something like:

RAG Chatbot ready. Type 'exit' to quit.

Ask a question: What did the Q3 architecture review say about database sharding?

Answer: The Q3 architecture review identified that the current monolithic Postgres
instance is approaching write throughput limits. The recommendation is to implement
application-level sharding on the `tenant_id` key by Q4, with a target of 8 shards.
The review also flagged that cross-shard JOINs will need to be handled at the
application layer using a scatter-gather pattern.

Sources:
  - architecture-review-q3.md (page N/A), score: 0.892
  - database-migration-plan.md (page N/A), score: 0.845
  - engineering-all-hands-notes.md (page N/A), score: 0.801
  - sharding-poc-results.pdf (page 3), score: 0.787

Citations are linked directly to source files. The scores reflect cosine similarity; above 0.75 is generally reliable, below 0.7 might be noise.

Extensions

Once the basic pipeline works, you can bolt on:

• Hybrid search: Combine dense vector search with sparse keyword search (BM25) for better recall on technical terms. Qdrant supports this natively—add a SparseEmbedding model and configure the query engine.
• Multi-user collections: Create per-user collections in Qdrant and route queries based on user ID. The free tier supports multiple collections within the 1GB limit.
• Streaming responses: Groq supports token streaming. Swap query_engine.query() for query_engine.query_stream() and print tokens as they arrive.
• PDF page-level citations: The current setup captures page labels when available. For scanned PDFs, add OCR with pytesseract before ingestion.
• Scheduled re-indexing: Wrap ingest.py in a cron job or a GitHub Action that watches a directory and pushes new documents nightly.

If you want to take this pattern into production—especially behind a customer’s firewall where Qdrant Cloud isn’t reachable—you’ll need the kind of architectural judgment that separates a weekend project from something an enterprise actually trusts. Our case study on deploying an LLM feature behind a Fortune 500 firewall walks through the real constraints you’ll hit.

Common Pitfalls

FAQ

Q: Why Qdrant instead of Chroma or Pinecone? Chroma is local-only by default and can be fiddly to persist. Pinecone’s free tier is limited to a single index. Qdrant Cloud gives you a fully managed, production-grade vector database with 1GB free—no credit card, no self-hosting, and it’s fast enough for real workloads.

Q: Can I use a different LLM? Yes. Swap Groq(model="mixtral-8x7b-32768") for any model Groq supports: Llama 3 70B, Gemma 7B, etc. Or drop in OpenAI from llama-index-llms-openai if you have API credits. The pipeline is LLM-agnostic.

Q: How do I handle a mix of languages in my documents? BGE-small-en-v1.5 is English-optimized. For multilingual documents, switch to BAAI/bge-m3 which supports 100+ languages. It’s larger and slower but available in FastEmbed.

Q: What if I want to deploy this as a Slack bot or web app? The query engine is stateless—wrap it in a FastAPI endpoint, add a simple auth layer, and you’ve got an API. For the full pattern of shipping a prototype from a messy customer problem in under a week, see how FDEs turn a messy problem into a shipped prototype.

Q: Will this work for hundreds of thousands of documents? Yes, but you’ll outgrow the Qdrant free tier. The architecture scales: chunking and embedding can be parallelized, Qdrant handles millions of vectors on paid plans, and Groq’s inference speed means retrieval latency stays low. The bottleneck becomes ingestion throughput, not query time.

Q: How does this compare to the Supabase-based RAG pipeline? We’ve covered a similar build using Supabase’s free vector store in this guide on building a RAG chatbot with Supabase and LlamaIndex. The Supabase approach gives you a full Postgres backend with row-level security—better if you need multi-tenancy. Qdrant is purpose-built for vectors and generally faster at pure ANN search.