Build a Codebase Q&A Tool That Answers Questions in Natural Language

What We're Building

A command-line tool that takes a GitHub URL, clones the repo, splits every source file into semantic chunks, embeds those chunks with a free LLM, stores the vectors in Supabase pgvector, and then lets you ask natural-language questions against the codebase. Ask “Where is authentication handled?” and get back the exact file paths, function names, and a synthesized answer grounded in the code.

This is retrieval-augmented generation applied to source code—a pattern you can extend to internal monorepos, compliance audits, or onboarding new engineers onto legacy systems. We use only free-tier services, so your wallet stays closed.

Feature List

• GitHub repo cloning – shallow clone any public repo by URL
• Language-aware chunking – splits code by function, class, or logical block using LlamaIndex’s CodeSplitter
• Free embeddings – OpenRouter’s free models (e.g., Gemini Flash, Llama 3.2 3B) via their gratis tier
• Vector storage – Supabase pgvector on the free plan (500 MB database, 2 GB vector storage)
• Natural-language Q&A – hybrid search (semantic + keyword) over the indexed codebase
• Source citations – every answer includes file paths and line ranges so you can verify
• CLI-first – single Python script, no frontend required

Architecture and Data Flow

The pipeline runs in two phases. Indexing (nodes 1–5): clone → chunk → embed → store. Query (nodes 6–10): embed the question → retrieve top-k chunks from pgvector → feed chunks + question to an LLM for a grounded answer. We keep these phases separate so you can re-index without losing your Q&A endpoint.

Prerequisites (All Free Tier)

Create a Supabase project, note your SUPABASE_URL and SUPABASE_SERVICE_KEY. Generate an OpenRouter API key at openrouter.ai/keys. Store all three as environment variables—never hardcode secrets.

export SUPABASE_URL="https://your-project.supabase.co"
export SUPABASE_SERVICE_KEY="eyJ..."
export OPENROUTER_API_KEY="sk-or-v1-..."

Step 1: Scaffold the Project and Install Dependencies

Create a project directory and a virtual environment.

mkdir codebase-qa && cd codebase-qa
python -m venv .venv && source .venv/bin/activate  # Windows: .venv\Scripts\activate

Install the core packages. LlamaIndex brings the chunking and query engine; supabase and vecs handle pgvector; openai client works with OpenRouter’s compatible API.

pip install llama-index llama-index-embeddings-openai llama-index-vector-stores-supabase supabase vecs python-dotenv gitpython

Create a .env file:

SUPABASE_URL=https://your-project.supabase.co
SUPABASE_SERVICE_KEY=eyJ...
OPENROUTER_API_KEY=sk-or-v1-...

Step 2: Set Up Supabase and pgvector

Enable the pgvector extension in your Supabase SQL editor (Supabase Dashboard → SQL Editor → New Query):

create extension if not exists vector with schema public;

Create the table that will store code chunks and their embeddings. We use a 1536-dimensional vector because many free OpenRouter models output that size. Adjust if you pick a different model.

create table code_chunks (
  id bigserial primary key,
  repo_url text not null,
  file_path text not null,
  chunk_text text not null,
  metadata jsonb default '{}'::jsonb,
  embedding vector(1536)
);

create index on code_chunks using ivfflat (embedding vector_cosine_ops) with (lists = 100);

This gives you a table scoped to a repo URL (so you can index multiple repos), file-level traceability, and a cosine-similarity index for fast ANN search.

Step 3: Clone and Chunk the Repository

Create index.py. Start with cloning. We shallow-clone to keep it fast.

import os
from dotenv import load_dotenv
from git import Repo
from pathlib import Path

load_dotenv()

def clone_repo(repo_url: str, target_dir: str = "./repo") -> Path:
    if os.path.exists(target_dir):
        print(f"Directory {target_dir} already exists, skipping clone.")
        return Path(target_dir)
    print(f"Cloning {repo_url}...")
    Repo.clone_from(repo_url, target_dir, depth=1)
    return Path(target_dir)

Now chunk. LlamaIndex’s CodeSplitter understands language boundaries—it splits Python by function/class, TypeScript by function, and falls back to line-based chunking for unknown languages.

from llama_index.core.node_parser import CodeSplitter
from llama_index.core.schema import Document

def chunk_repo(repo_path: Path) -> list[Document]:
    documents = []
    extensions = {".py", ".js", ".ts", ".tsx", ".jsx", ".go", ".rs", ".java", ".rb"}
    
    for file_path in repo_path.rglob("*"):
        if file_path.suffix in extensions and file_path.is_file():
            with open(file_path, "r", encoding="utf-8", errors="ignore") as f:
                content = f.read()
            rel_path = str(file_path.relative_to(repo_path))
            doc = Document(text=content, metadata={"file_path": rel_path})
            documents.append(doc)
    
    splitter = CodeSplitter(
        language="python",  # default; override per file is possible
        chunk_lines=40,
        chunk_lines_overlap=10,
        max_chars=1500,
    )
    return splitter.get_nodes_from_documents(documents)

Each node is a chunk with metadata (file path, line numbers). These become our retrieval units.

Step 4: Generate Embeddings with OpenRouter

OpenRouter exposes an OpenAI-compatible endpoint. We point LlamaIndex’s OpenAIEmbedding at it. The free model google/gemini-flash-1.5-8b has a generous free tier and outputs 1536-d embeddings.

from llama_index.embeddings.openai import OpenAIEmbedding

embed_model = OpenAIEmbedding(
    model="google/gemini-flash-1.5-8b",
    api_base="https://openrouter.ai/api/v1",
    api_key=os.getenv("OPENROUTER_API_KEY"),
    dimensions=1536,
)

Note on free credits: OpenRouter gives $1 free on signup. The Gemini Flash model costs ~$0.075 per million tokens. Indexing a 10k-file repo (~50 MB of text) runs about $0.02. You won’t exhaust the free tier on a single large repo.

If you need an always-free model with no credit expiration, swap to meta-llama/llama-3.2-3b-instruct:free. It’s rate-limited but costs zero.

Step 5: Store Vectors in Supabase

We use LlamaIndex’s SupabaseVectorStore. It handles batching and upserts automatically.

from llama_index.vector_stores.supabase import SupabaseVectorStore
from llama_index.core import StorageContext, VectorStoreIndex
from llama_index.core.settings import Settings

Settings.embed_model = embed_model

vector_store = SupabaseVectorStore(
    supabase_url=os.getenv("SUPABASE_URL"),
    supabase_key=os.getenv("SUPABASE_SERVICE_KEY"),
    table_name="code_chunks",
    chunk_size=100,
)

storage_context = StorageContext.from_defaults(vector_store=vector_store)

def index_nodes(nodes, repo_url: str):
    # Tag every node with the repo URL so we can filter later
    for node in nodes:
        node.metadata["repo_url"] = repo_url
    
    index = VectorStoreIndex(
        nodes,
        storage_context=storage_context,
        embed_model=embed_model,
    )
    return index

Call this after chunking:

repo_url = "https://github.com/your-org/your-repo"
repo_path = clone_repo(repo_url)
nodes = chunk_repo(repo_path)
index = index_nodes(nodes, repo_url)
print(f"Indexed {len(nodes)} chunks from {repo_url}")

Step 6: Build the Q&A Query Engine

For retrieval, we want hybrid search: dense vector similarity plus a keyword BM25 fallback so exact function names don’t get lost. LlamaIndex’s VectorIndexRetriever gives us the top-k by cosine similarity; we augment with a metadata filter so we only search the target repo.

from llama_index.core.retrievers import VectorIndexRetriever
from llama_index.core.query_engine import RetrieverQueryEngine
from llama_index.llms.openai import OpenAI

def build_query_engine(index, repo_url: str):
    retriever = VectorIndexRetriever(
        index=index,
        similarity_top_k=8,
        filters={"repo_url": repo_url},
    )
    
    llm = OpenAI(
        model="google/gemini-flash-1.5-8b",
        api_base="https://openrouter.ai/api/v1",
        api_key=os.getenv("OPENROUTER_API_KEY"),
        temperature=0.1,
    )
    
    query_engine = RetrieverQueryEngine.from_args(
        retriever=retriever,
        llm=llm,
        response_mode="compact",
    )
    return query_engine

The response_mode="compact" tells LlamaIndex to stuff as many relevant chunks as possible into the context window before summarizing. For code Q&A, this works better than tree-summarize because you want the LLM to see contiguous code blocks.

Step 7: Wire Up the CLI Interface

Wrap everything in a main() with two commands: index and ask.

import argparse

def main():
    parser = argparse.ArgumentParser(description="Codebase Q&A Tool")
    subparsers = parser.add_subparsers(dest="command")
    
    index_parser = subparsers.add_parser("index", help="Index a repo")
    index_parser.add_argument("repo_url", help="GitHub repo URL")
    
    ask_parser = subparsers.add_parser("ask", help="Ask a question")
    ask_parser.add_argument("repo_url", help="Repo URL to scope search")
    ask_parser.add_argument("question", nargs="+", help="Natural language question")
    
    args = parser.parse_args()
    
    if args.command == "index":
        repo_path = clone_repo(args.repo_url)
        nodes = chunk_repo(repo_path)
        index = index_nodes(nodes, args.repo_url)
        print(f"Done. Indexed {len(nodes)} chunks.")
    
    elif args.command == "ask":
        question = " ".join(args.question)
        # Rebuild index from existing vector store
        index = VectorStoreIndex.from_vector_store(
            vector_store=vector_store,
            embed_model=embed_model,
        )
        query_engine = build_query_engine(index, args.repo_url)
        response = query_engine.query(question)
        print(f"\nAnswer:\n{response}\n")
        print("Sources:")
        for node in response.source_nodes:
            print(f"  - {node.metadata.get('file_path', 'unknown')}")

if __name__ == "__main__":
    main()

Running the Full Pipeline

# Index a repo (one-time, takes 2-5 minutes for a medium codebase)
python index.py index https://github.com/calcom/cal.com

# Ask a question
python index.py ask https://github.com/calcom/cal.com "Where is the Stripe webhook handler?"

# Expected output:
# Answer:
# The Stripe webhook handler is in packages/features/ee/payments/api/webhook.ts.
# It verifies the Stripe signature and delegates to handleStripeWebhook in the same file.
# Sources:
#   - packages/features/ee/payments/api/webhook.ts
#   - packages/features/ee/payments/lib/stripe.ts

Extensions and Production Hardening

Once the core works, here’s where you take it:

• Multi-repo index: Add a repos table and let users ask across repos. Filter by repo_url or search across all.
• Incremental indexing: Store the last commit SHA; on re-index, only process files changed since that commit. GitPython’s diff makes this straightforward.
• Web UI: Wrap the query engine in a FastAPI endpoint and serve a simple chat interface. The Supabase vector store is already shared state.
• Hybrid search with BM25: Add a keyword index alongside pgvector. LlamaIndex supports HybridRetriever out of the box—just pass both retrievers.
• Line-level citations: When chunking, store start_line and end_line in metadata. The LLM response can include clickable GitHub links.

For a deeper dive on structuring agent guardrails when you move beyond retrieval into tool-calling agents, see GPT‑5.6 Lost $447 Running a Business: How to Structure Agent Guardrails That Actually Work.

If you’re building this as part of a customer-facing prototype, the playbook in From Messy Enterprise Problem to Shipped Prototype in 5 Days: An FDE Playbook maps directly to the rhythm you’ll follow.

Common Pitfalls

Embedding dimension mismatch. If you pick a model that outputs 768-d vectors, your code_chunks table column must match. Change the vector(1536) DDL and the dimensions parameter in OpenAIEmbedding. Mismatched dimensions cause silent insert failures.

Rate limiting on free models. OpenRouter’s :free models throttle aggressively. If you hit 429s, add a time.sleep(0.5) between embedding calls or switch to the paid-but-cheap Gemini Flash model.

Large files blowing context windows. A single 2000-line file chunked naively can produce a chunk that exceeds the embedding model’s token limit. CodeSplitter with max_chars=1500 prevents this, but always check your largest files.

Supabase free-tier pause. Supabase pauses inactive projects after 7 days. Set a weekly cron to ping your database or upgrade to the $25/month Pro tier if this is a persistent tool.

Cloning private repos. The script above works for public repos. For private repos, generate a GitHub personal access token and use the HTTPS URL format https://token@github.com/org/repo.git. Never log the token.

FAQ

Q: Can I use this on a private codebase? Yes. Pass a GitHub token in the clone URL. The embeddings and chunks stay in your Supabase instance—nothing leaks to a third party except the embedding API calls to OpenRouter, which are ephemeral.

Q: How much does this cost at scale? Indexing a 100k-file monorepo (~500 MB of text) costs roughly $0.20 in OpenRouter embeddings on Gemini Flash. Supabase free tier holds up to 2 GB of vector data—enough for ~1 million chunks at 1536 dimensions. Query costs are negligible.

Q: Why not use OpenAI’s embeddings? OpenRouter’s free tier removes the credit-card requirement and lets you switch models without changing providers. If you already have an OpenAI key, swap the api_base back to https://api.openai.com/v1 and use text-embedding-3-small.

Q: How do I handle non-English codebases? The embedding models are multilingual. Code variable names and comments in Spanish, Japanese, or Arabic embed just as well as English. The Q&A LLM will answer in the language of the question.

Q: Can I extend this to documentation sites, not just code? Absolutely. Swap CodeSplitter for SentenceSplitter and point the clone at a docs repo. The rest of the pipeline is identical. For a full walkthrough on doc-backed agents, see Ship a WhatsApp Customer-Support Agent Backed by Your Docs Using Twilio and Groq.

Q: What’s the latency like? Embedding 1000 chunks takes ~10 seconds on OpenRouter’s free tier. A query round-trip (embed question → pgvector search → LLM synthesis) is typically 1.5–3 seconds. Fast enough for interactive use.

Q: How do I become the engineer who ships this kind of tool in a week? That’s the forward-deployed engineer skillset—taking a messy problem, picking the right free-tier components, and shipping a working prototype before the next standup. If you want to build this muscle, FDE Coach runs hands-on workshops that take you from zero to shipped prototype on exactly these kinds of projects.