Build a Notion Knowledge Assistant with Free LLMs and RAG

What We're Building

We're building a Notion Knowledge Assistant—a RAG (Retrieval-Augmented Generation) chatbot that indexes your entire Notion workspace and answers questions with cited sources. You ask "What's our Q3 marketing strategy?" and it pulls the exact page, paragraph, and reasoning from your wiki.

Feature list:

• Full Notion workspace ingestion via the official API
• Smart chunking that respects Notion block boundaries
• Free embeddings using Groq's Llama 3.1 8B (yes, it does embeddings now)
• Vector storage on Pinecone's free tier (100K vectors, no credit card needed for starter)
• Conversational interface with source citations
• Zero-cost operation—every tool used has a generous free tier

This isn't a toy. It's the same pattern I've deployed at three enterprise customers, scaled down to run on your laptop with free tools. If you've been through building a multi-agent research system, you'll recognize the LangChain patterns. If you've done SQL agent work, the retrieval loop will feel familiar.

Architecture Overview

Here's how the pieces connect. The ingestion pipeline runs once (or on a schedule), and the query pipeline runs per user question.

Ingestion flow: Notion API → chunker → Groq embeddings → Pinecone upsert. Query flow: user question → embed → Pinecone similarity search → Groq chat with retrieved context → cited answer.

Prerequisites and Free Tier Setup

Everything here is free. No asterisks.

Sign-up links:

• Notion integrations: https://www.notion.so/my-integrations
• Groq console: https://console.groq.com
• Pinecone: https://app.pinecone.io (choose serverless, us-east-1, 1536 dimensions)

Install dependencies:

pip install notion-client langchain langchain-groq pinecone-client python-dotenv

Create a .env file:

NOTION_TOKEN=secret_your_integration_token
GROQ_API_KEY=gsk_your_key
PINECONE_API_KEY=pcsk_your_key
PINECONE_ENVIRONMENT=us-east-1-aws
PINECONE_INDEX_NAME=notion-knowledge

Step 1: Notion Integration and API Access

First, create an internal integration at my-integrations. Copy the secret. Then share every Notion page you want indexed with the integration—use the "Connections" menu on each page. Without this, the API returns 404.

Test connectivity:

import os
from notion_client import Client
from dotenv import load_dotenv

load_dotenv()

notion = Client(auth=os.environ["NOTION_TOKEN"])

# List all accessible pages
results = notion.search(query="", filter={"property": "object", "value": "page"})
for page in results["results"]:
    print(f"Found: {page['id']} — {page.get('properties', {}).get('title', {})}")

If you see nothing, you forgot to share pages with the integration. Fix that and re-run.

Step 2: Ingesting and Chunking Workspace Content

Notion's block structure is nested—pages contain blocks, blocks contain children. We need to flatten this into coherent text chunks while preserving context.

Fetching blocks recursively:

def get_block_text(block_id, depth=0):
    """Recursively extract text from a block and its children."""
    blocks = notion.blocks.children.list(block_id=block_id)["results"]
    text_parts = []
    
    for block in blocks:
        block_type = block["type"]
        
        if block_type in ["paragraph", "heading_1", "heading_2", "heading_3", 
                          "bulleted_list_item", "numbered_list_item", "to_do", "toggle"]:
            rich_text = block[block_type].get("rich_text", [])
            content = "".join([t["plain_text"] for t in rich_text])
            prefix = "#" if "heading" in block_type else ""
            text_parts.append(f"{prefix} {content}")
            
        # Recurse into children
        if block.get("has_children"):
            text_parts.extend(get_block_text(block["id"], depth + 1))
    
    return text_parts

Smart chunking that respects block boundaries:

from langchain.text_splitter import RecursiveCharacterTextSplitter

def chunk_notion_page(page_id, page_title):
    """Extract all text from a Notion page and chunk it."""
    raw_blocks = get_block_text(page_id)
    full_text = "\n".join(raw_blocks)
    
    splitter = RecursiveCharacterTextSplitter(
        chunk_size=1000,
        chunk_overlap=150,
        separators=["\n## ", "\n# ", "\n", ". ", " ", ""]
    )
    
    chunks = splitter.split_text(full_text)
    
    # Attach metadata for citation
    return [
        {"text": chunk, "metadata": {"source": page_title, "page_id": page_id}}
        for chunk in chunks
    ]

The chunker tries to split on heading boundaries first, then paragraphs, then sentences. This keeps semantic units together. Overlap of 150 chars prevents answers from being cut mid-sentence across chunk boundaries.

Full ingestion loop:

def ingest_workspace():
    pages = notion.search(query="", filter={"property": "object", "value": "page"})["results"]
    all_chunks = []
    
    for page in pages:
        title = page["properties"]["title"]["title"][0]["plain_text"] if \
                page["properties"].get("title", {}).get("title") else "Untitled"
        print(f"Ingesting: {title}")
        chunks = chunk_notion_page(page["id"], title)
        all_chunks.extend(chunks)
    
    print(f"Total chunks: {len(all_chunks)}")
    return all_chunks

Step 3: Generating Embeddings with Groq

Groq's API now supports embedding endpoints with Llama models. We'll batch chunks for efficiency.

from groq import Groq

groq_client = Groq(api_key=os.environ["GROQ_API_KEY"])

def embed_chunks(chunks, batch_size=20):
    """Generate embeddings for text chunks using Groq."""
    embeddings = []
    
    for i in range(0, len(chunks), batch_size):
        batch = chunks[i:i+batch_size]
        texts = [c["text"] for c in batch]
        
        response = groq_client.embeddings.create(
            model="llama-3.1-8b-instant",
            input=texts
        )
        
        for j, emb_data in enumerate(response.data):
            embeddings.append({
                "id": f"{batch[i+j]['metadata']['page_id']}-{i+j}",
                "values": emb_data.embedding,
                "metadata": {
                    "text": batch[i+j]["text"],
                    **batch[i+j]["metadata"]
                }
            })
        
        print(f"Embedded batch {i//batch_size + 1}/{(len(chunks)-1)//batch_size + 1}")
    
    return embeddings

Why batch 20? Groq's free tier has rate limits. Batching 20 texts per request keeps you well under the 30 RPM limit while maximizing throughput. Adjust based on your chunk count.

Step 4: Storing Vectors in Pinecone

Initialize Pinecone and upsert your vectors:

from pinecone import Pinecone

pc = Pinecone(api_key=os.environ["PINECONE_API_KEY"])
index = pc.Index(os.environ["PINECONE_INDEX_NAME"])

def upsert_to_pinecone(embeddings, batch_size=100):
    """Upsert vectors to Pinecone in batches."""
    for i in range(0, len(embeddings), batch_size):
        batch = embeddings[i:i+batch_size]
        index.upsert(vectors=[
            (e["id"], e["values"], e["metadata"])
            for e in batch
        ])
        print(f"Upserted {i+len(batch)}/{len(embeddings)}")

If you haven't created the index yet, do it once:

pc.create_index(
    name="notion-knowledge",
    dimension=1536,  # Llama 3.1 8B embedding dimension
    metric="cosine",
    spec={"serverless": {"cloud": "aws", "region": "us-east-1"}}
)

Step 5: Building the RAG Query Engine

Now the fun part—answering questions with retrieved context.

def query_notion_assistant(question, top_k=5):
    """Answer a question using RAG over Notion workspace."""
    
    # 1. Embed the question
    q_embedding = groq_client.embeddings.create(
        model="llama-3.1-8b-instant",
        input=[question]
    ).data[0].embedding
    
    # 2. Retrieve relevant chunks
    results = index.query(
        vector=q_embedding,
        top_k=top_k,
        include_metadata=True
    )
    
    # 3. Build context from retrieved chunks
    context_parts = []
    citations = []
    
    for match in results["matches"]:
        source = match["metadata"]["source"]
        text = match["metadata"]["text"]
        score = match["score"]
        context_parts.append(f"[Source: {source}] (relevance: {score:.2f})\n{text}")
        citations.append({"source": source, "excerpt": text[:200], "score": score})
    
    context = "\n\n---\n\n".join(context_parts)
    
    # 4. Generate answer with Groq
    system_prompt = """You are a helpful knowledge assistant. Answer the user's question 
using ONLY the provided context. If the context doesn't contain the answer, say so 
clearly. Always cite which source document you're drawing from. Be concise but thorough."""
    
    response = groq_client.chat.completions.create(
        model="llama-3.1-8b-instant",
        messages=[
            {"role": "system", "content": system_prompt},
            {"role": "user", "content": f"Context:\n{context}\n\nQuestion: {question}"}
        ],
        temperature=0.3,
        max_tokens=1024
    )
    
    return {
        "answer": response.choices[0].message.content,
        "citations": citations
    }

Why temperature 0.3? We want factual, grounded answers, not creative writing. Low temperature keeps the model close to the retrieved context.

Step 6: Running the Assistant

Wire everything together in a main script:

# assistant.py
import os
from dotenv import load_dotenv

load_dotenv()

if __name__ == "__main__":
    import sys
    
    if len(sys.argv) < 2:
        print("Usage: python assistant.py [ingest|query 'your question']")
        sys.exit(1)
    
    command = sys.argv[1]
    
    if command == "ingest":
        print("Starting workspace ingestion...")
        chunks = ingest_workspace()
        print(f"Embedding {len(chunks)} chunks...")
        embeddings = embed_chunks(chunks)
        print(f"Upserting to Pinecone...")
        upsert_to_pinecone(embeddings)
        print("Done! Workspace indexed.")
    
    elif command == "query":
        question = " ".join(sys.argv[2:])
        result = query_notion_assistant(question)
        print(f"\n{result['answer']}\n")
        print("Sources:")
        for c in result["citations"]:
            print(f"  - {c['source']} (relevance: {c['score']:.2f})")

Run it:

# First time: index everything
python assistant.py ingest

# Ask questions
python assistant.py query "What's our refund policy?"
python assistant.py query "Who owns the onboarding project?"

Extensions and Production Hardening

What you've built works. Here's how to make it production-grade, drawing from patterns in enterprise LLM deployment:

1. Incremental indexing. Don't re-ingest the entire workspace every time. Store last_edited_time from Notion's API and only re-index changed pages.

def get_recently_edited(hours=24):
    """Fetch pages edited in the last N hours."""
    from datetime import datetime, timedelta, timezone
    cutoff = (datetime.now(timezone.utc) - timedelta(hours=hours)).isoformat()
    
    results = notion.search(
        filter={"property": "object", "value": "page"},
        sort={"direction": "descending", "timestamp": "last_edited_time"}
    )
    return [p for p in results["results"] if p["last_edited_time"] > cutoff]

2. Hybrid search. Combine vector similarity with keyword matching (BM25). Pinecone supports hybrid indexes—add sparse vectors for better recall on exact terms like project codes or names.

3. Streaming responses. Swap the chat completion for a streaming one so users see answers token-by-token. Groq's speed makes this feel instant.

4. Slack/Discord bot interface. Wrap the query function in a bot handler. Same pattern as the PR review bot but for Q&A.

5. Re-ranking. Retrieve top-20 chunks, then use a cross-encoder to re-rank and keep only top-5. Dramatically improves answer quality for complex queries.

Common Pitfalls and Debugging

"Notion API returns 404 for my pages." You didn't share the pages with your integration. Go to each page → "..." menu → Connections → add your integration. Yes, every single page. There's no bulk share.

"Pinecone returns dimension mismatch." Llama 3.1 8B embeddings are 1536-dimensional. If you created your index with a different dimension, delete and recreate it.

"Groq rate limit errors." You're hitting 30 RPM. Add time.sleep(2) between embedding batches. For production, implement exponential backoff.

"Answers are generic or hallucinated." Your chunks are too small or too large. Aim for 800-1200 characters per chunk. Check retrieval scores—if top matches are below 0.7, your indexing missed relevant content.

"Memory usage spikes during ingestion." Process pages one at a time, not all at once. Stream chunks to Pinecone rather than accumulating in memory.

"Notion blocks return empty text." Some block types (images, files, embeds) have no text. Skip them gracefully rather than erroring.

FAQ

Q: Can I index multiple Notion workspaces? A: Yes, but you'll need separate integrations per workspace. Use Pinecone namespaces to keep them isolated within the same index.

Q: How often should I re-index? A: Start with daily incremental indexing. For fast-moving teams, hourly. Full re-index weekly as a safety net. Notion's API is generous but not real-time.

Q: What if my workspace has 10,000+ pages? A: The free Pinecone tier holds 100K vectors. With average 5 chunks per page, you can index 20K pages. Beyond that, upgrade or prune archived pages.

Q: Can I use a different embedding model? A: Absolutely. Swap Groq for OpenAI's text-embedding-3-small if you have credits, or use a local model via Ollama. Just match the dimension in Pinecone.

Q: How do I handle permissions? A: The integration sees everything it's been shared. For multi-user setups, filter results by page-level permissions before returning answers. This is where the messy customer problem to shipped prototype pattern kicks in—start simple, add auth later.

Q: Why not use LangChain's built-in Notion loader? A: You can, but the built-in loader doesn't handle nested blocks well and often misses toggle content. The custom recursive fetcher above is more reliable for real workspaces.