Deploy a RAG Chatbot Over Your PDFs and Notes with Qdrant Free Tier & Groq

What We're Building

We’re shipping a local-first RAG (Retrieval-Augmented Generation) chatbot that answers questions from a folder of PDFs and markdown notes. It returns answers with source citations—so you know exactly which file and page the information came from. The entire stack runs on free-tier tools: Groq for fast LLM inference, Qdrant’s free cloud tier for vector storage, Hugging Face Inference API for embeddings, LlamaIndex as the orchestration framework, and Streamlit for the UI.

Feature list:

• Ingest a local folder of PDFs and .md files
• Chunk documents intelligently with LlamaIndex
• Generate embeddings via Hugging Face Inference API (free tier)
• Store vectors in Qdrant’s free cloud cluster
• Query with Groq’s blazing-fast LLMs (Mixtral, Llama 3, etc.)
• Display answers with clickable source citations in Streamlit
• Fully local except for API calls—your files never leave your machine except as embeddings

If you’ve built a WhatsApp customer-support agent before, you’ll find the retrieval pattern familiar—this time we’re pointing it at your personal knowledge base.

Architecture Overview

Before we write code, let’s map how data flows from your files to the answer on screen.

The ingestion pipeline (left side) loads files, chunks them, embeds each chunk via Hugging Face, and upserts into Qdrant. The query pipeline (right side) embeds the user’s question, retrieves the top-k semantically similar chunks from Qdrant, packages them as context for Groq, and streams the answer back to Streamlit.

Prerequisites

Everything here has a generous free tier. Grab these before you start:

1. Python 3.10+ – python.org/downloads
2. Groq API key – Sign up at console.groq.com. Free tier gives you rate-limited access to Mixtral, Llama 3, and Gemma models.
3. Qdrant Cloud free tier – Create a cluster at cloud.qdrant.io. The free tier includes 1GB of storage and a single node. Copy your cluster URL and API key.
4. Hugging Face API token – Go to huggingface.co/settings/tokens and create a read-access token. The free Inference API lets you call embedding models like sentence-transformers/all-MiniLM-L6-v2 without deploying anything.
5. Your documents – A folder containing PDFs and/or .md files. Start with 5–10 files to keep initial indexing fast.

Step 1: Project Setup and Dependencies

Create a project folder and a virtual environment:

mkdir rag-chatbot && cd rag-chatbot
python -m venv venv
source venv/bin/activate  # Windows: venv\Scripts\activate

Install the core packages:

pip install llama-index llama-index-vector-stores-qdrant llama-index-embeddings-huggingface
pip install llama-index-llms-groq streamlit qdrant-client python-dotenv

Create a .env file in the project root:

GROQ_API_KEY=gsk_your_groq_key_here
QDRANT_URL=https://your-cluster-id.us-east-1-0.aws.cloud.qdrant.io:6333
QDRANT_API_KEY=your_qdrant_api_key_here
HF_TOKEN=hf_your_huggingface_token_here

Create app.py—that’s where the whole application lives. Let’s build it section by section.

Step 2: Loading Documents with LlamaIndex

LlamaIndex’s SimpleDirectoryReader handles PDFs and markdown files out of the box. It auto-detects file types and extracts text.

# app.py
import os
from dotenv import load_dotenv
from llama_index.core import SimpleDirectoryReader, Settings

load_dotenv()

DATA_DIR = "./data"  # Put your PDFs and .md files here

def load_documents():
    if not os.path.exists(DATA_DIR):
        os.makedirs(DATA_DIR)
        print(f"Created {DATA_DIR}/ — add your PDFs and .md files there.")
        return []
    reader = SimpleDirectoryReader(DATA_DIR, recursive=True)
    documents = reader.load_data()
    print(f"Loaded {len(documents)} documents.")
    return documents

The recursive=True flag picks up files in subdirectories. Each document object carries both the text content and metadata (file name, page number for PDFs).

Step 3: Generating Embeddings with Hugging Face Inference API

We’ll use sentence-transformers/all-MiniLM-L6-v2 via the free Inference API. This model produces 384-dimensional embeddings—compact enough to keep Qdrant storage low, performant enough for semantic search over notes.

LlamaIndex has a dedicated Hugging Face Inference API embedding class:

from llama_index.embeddings.huggingface import HuggingFaceInferenceAPIEmbedding

def setup_embed_model():
    embed_model = HuggingFaceInferenceAPIEmbedding(
        model_name="sentence-transformers/all-MiniLM-L6-v2",
        token=os.getenv("HF_TOKEN"),
    )
    Settings.embed_model = embed_model
    return embed_model

Settings.embed_model is LlamaIndex’s global configuration hook. Once set, every component that needs embeddings picks it up automatically.

Step 4: Indexing with Qdrant Free Tier

Qdrant’s free tier gives you a managed vector database. We’ll create a collection and index our documents there.

from llama_index.vector_stores.qdrant import QdrantVectorStore
from llama_index.core import VectorStoreIndex, StorageContext
from qdrant_client import QdrantClient

def create_index(documents, embed_model):
    client = QdrantClient(
        url=os.getenv("QDRANT_URL"),
        api_key=os.getenv("QDRANT_API_KEY"),
    )
    collection_name = "my_notes"

    vector_store = QdrantVectorStore(
        client=client,
        collection_name=collection_name,
    )
    storage_context = StorageContext.from_defaults(vector_store=vector_store)

    index = VectorStoreIndex.from_documents(
        documents,
        storage_context=storage_context,
        embed_model=embed_model,
    )
    print(f"Indexed {len(documents)} documents into Qdrant collection '{collection_name}'.")
    return index

Important: Qdrant’s free tier does not support deleting collections via the API in some configurations. If you want to re-index, use client.delete_collection(collection_name) before creating the vector store, or increment the collection name.

Step 5: Querying with Groq LLM

Groq’s LPU hardware delivers absurdly fast token generation. We’ll use llama-3.1-8b-instant for quick, free answers. Swap to mixtral-8x7b-32768 if you need heavier reasoning.

from llama_index.llms.groq import Groq

def setup_llm():
    llm = Groq(
        model="llama-3.1-8b-instant",
        api_key=os.getenv("GROQ_API_KEY"),
        temperature=0.2,
    )
    Settings.llm = llm
    return llm

def build_query_engine(index):
    return index.as_query_engine(
        similarity_top_k=4,
        response_mode="compact",
    )

similarity_top_k=4 retrieves the four most relevant chunks. response_mode="compact" packs them into the context window efficiently—fewer tokens, lower latency. The engine automatically cites sources in the response.

Step 6: Building the Streamlit UI

We want a clean chat interface with source citations. Streamlit’s chat_message and chat_input widgets make this straightforward.

import streamlit as st

def run_ui(query_engine):
    st.set_page_config(page_title="RAG Chatbot", page_icon="📚")
    st.title("📚 RAG Over Your Notes")
    st.caption("Ask questions about your PDFs and markdown files. Sources are cited inline.")

    if "messages" not in st.session_state:
        st.session_state.messages = []

    for msg in st.session_state.messages:
        with st.chat_message(msg["role"]):
            st.markdown(msg["content"])

    if prompt := st.chat_input("What do you want to know?"):
        st.session_state.messages.append({"role": "user", "content": prompt})
        with st.chat_message("user"):
            st.markdown(prompt)

        with st.chat_message("assistant"):
            with st.spinner("Searching your notes..."):
                response = query_engine.query(prompt)
            st.markdown(response.response)
            if hasattr(response, "source_nodes") and response.source_nodes:
                with st.expander("📎 Sources"):
                    for i, node in enumerate(response.source_nodes, 1):
                        file_name = node.metadata.get("file_name", "Unknown")
                        page = node.metadata.get("page_label", "N/A")
                        snippet = node.text[:300].replace("\n", " ")
                        st.markdown(f"**{i}. {file_name}** (page {page})")
                        st.caption(snippet)

        st.session_state.messages.append({"role": "assistant", "content": response.response})

The source expander shows which files and pages contributed to the answer. The snippet preview lets the user gauge relevance without opening the file.

Step 7: Running the Full Application

Wire everything together in app.py:

def main():
    setup_embed_model()
    setup_llm()

    documents = load_documents()
    if not documents:
        st.warning("No documents found. Add PDFs or .md files to the ./data folder and restart.")
        return

    index = create_index(documents, Settings.embed_model)
    query_engine = build_query_engine(index)
    run_ui(query_engine)

if __name__ == "__main__":
    main()

Run it:

streamlit run app.py

On first run, the app indexes your documents—this may take a minute depending on file count and Hugging Face API latency. Subsequent runs can skip re-indexing by persisting the Qdrant collection. Add a check: if the collection already exists, load from Qdrant instead of re-indexing.

Lazy-loading version:

def get_or_create_index(documents, embed_model):
    client = QdrantClient(
        url=os.getenv("QDRANT_URL"),
        api_key=os.getenv("QDRANT_API_KEY"),
    )
    collection_name = "my_notes"

    # Check if collection exists
    collections = client.get_collections().collections
    exists = any(c.name == collection_name for c in collections)

    vector_store = QdrantVectorStore(client=client, collection_name=collection_name)

    if exists:
        print("Collection exists, loading existing index...")
        index = VectorStoreIndex.from_vector_store(
            vector_store,
            embed_model=embed_model,
        )
    else:
        print("Creating new index...")
        storage_context = StorageContext.from_defaults(vector_store=vector_store)
        index = VectorStoreIndex.from_documents(
            documents,
            storage_context=storage_context,
            embed_model=embed_model,
        )
    return index

This avoids re-embedding and re-uploading every time you restart.

Sensible Extensions

Once the core loop works, you can layer on these improvements:

• Hybrid search: Combine keyword (BM25) and semantic search by enabling Qdrant’s sparse_vector support. LlamaIndex supports hybrid retrieval with a QueryFusionRetriever.
• Multi-user sessions: Add a simple password gate in Streamlit and scope Qdrant collections per user (e.g., notes_{user_id}).
• Chat history: Feed the last N messages into the query engine as chat_history for follow-up questions. LlamaIndex’s CondenseQuestionChatEngine handles this cleanly.
• More file types: SimpleDirectoryReader also handles .txt, .csv, .docx, and .epub. Drop them in and they’ll be ingested.
• Local embeddings: If you hit Hugging Face rate limits, switch to llama-index-embeddings-ollama and run nomic-embed-text locally. Slightly higher setup cost, zero API dependency.

If you’re enjoying this pattern, you might also like our guide on building a Gmail AI triage agent that drafts replies using the same Groq free tier—same LLM, different retrieval surface.

Common Pitfalls

1. Hugging Face 503 errors on first call The free Inference API cold-starts models. The first embedding call may time out. Retry logic helps:

import time
def embed_with_retry(embed_model, texts, max_retries=3):
    for attempt in range(max_retries):
        try:
            return embed_model.get_text_embedding_batch(texts)
        except Exception as e:
            if attempt == max_retries - 1:
                raise
            time.sleep(2 ** attempt)

2. Qdrant free-tier rate limits The free tier throttles writes. Index large document sets in batches of 50 chunks with a short time.sleep(1) between batches.

3. PDF parsing quality SimpleDirectoryReader uses PyPDF2 by default, which struggles with complex layouts. For better extraction, install pymupdf (pip install pymupdf) and LlamaIndex will use it automatically.

4. Groq context window limits llama-3.1-8b-instant has a 128K context window, but packing too many chunks dilutes answer quality. Stick to similarity_top_k=4 unless you have a specific reason to go higher.

5. Environment variables not loading Streamlit runs app.py from its own process. Ensure load_dotenv() is called at the top of the file, before any client initialization.

The skills you’re building here—stitching retrieval, LLMs, and a clean UI—are exactly what make a Forward Deployed Engineer effective in the field. Shipping a working prototype in an afternoon is the name of the game.

FAQ

Q: How many documents can the free tier handle? A: Qdrant’s free tier gives you 1GB of vector storage. With 384-dimensional embeddings, that’s roughly 2–3 million chunks. For a personal notes use case, you’ll likely never hit the limit.

Q: Can I use a local LLM instead of Groq? A: Absolutely. Swap Groq for Ollama (pip install llama-index-llms-ollama) and point it at a local model. You’ll lose Groq’s speed but gain full offline capability.

Q: Why Hugging Face Inference API instead of local embeddings? A: It’s zero-setup—no GPU required, no model downloads. The free tier is generous for personal use. If you need offline or higher throughput, Ollama’s embedding endpoint is a drop-in replacement.

Q: How do I update the index when I add new files? A: Use LlamaIndex’s insert() method on the index object, or re-run the ingestion script pointing only at new files. For a production setup, look into LlamaIndex’s IngestionPipeline with document tracking.

Q: Is my data secure? A: Your original files stay on your machine. Only embedding vectors (arrays of floats) are sent to Qdrant and Hugging Face. The text chunks are sent to Groq at query time. Review each provider’s privacy policy if you’re handling sensitive data.

Q: What’s a good next project after this? A: Try building a resume tailoring agent that rewrites your CV for specific job descriptions—it uses a similar retrieval pattern but flips the objective toward generation rather than Q&A.