Build a Codebase Q&A Bot with Gemini RAG and LlamaIndex for Free

What We're Building

We're building a local CLI tool that turns any codebase into a searchable knowledge base. Point it at a directory, and it recursively ingests every source file, chunks them intelligently, stores embeddings in ChromaDB, and lets you fire natural-language questions at it. Answers come from Google Gemini's free tier, grounded in your actual code.

Feature list:

• Recursive ingestion of a local repo (respects .gitignore patterns)
• Language-aware chunking via LlamaIndex's CodeSplitter
• Persistent vector storage with ChromaDB (no re-indexing on restart)
• Gemini gemini-1.5-flash free-tier model for embeddings and generation
• Interactive CLI with session history
• Token usage tracking so you don't accidentally blow through free-tier limits

If you've ever inherited a sprawling monorepo at 2 AM during an incident, you know why this matters. Instead of grep roulette, you ask "Which service owns the payment retry logic?" and get a direct answer with file paths and line references.

Architecture Overview

Here's how the pieces fit together. The ingestion pipeline reads files, splits them into semantic chunks, and pushes vectors into ChromaDB. At query time, the retriever pulls relevant chunks, and Gemini synthesizes an answer grounded in those chunks.

The flow is intentionally linear. Ingestion runs once (or on-demand when the repo changes). Queries hit ChromaDB directly—no network calls except to Google's API for embeddings and generation. ChromaDB stores vectors on disk in ./chroma_db, so you can kill the process and pick up where you left off.

Prerequisites and Free-Tier Setup

Everything here runs on free tiers. No credit card required for the core flow, though Google asks for one to verify you're not a bot.

What you need:

1. Python 3.10+ — python.org/downloads
2. Google Gemini API key — Grab one at aistudio.google.com/apikey. The free tier gives you 15 requests per minute for gemini-1.5-flash. That's plenty for a solo dev tool.
3. Git (optional) — if you want to clone a repo to test against.

Free-tier limits to know:

• gemini-1.5-flash: 15 RPM, 1 million tokens per minute, 1,500 requests per day. Embeddings count toward the same quota.
• ChromaDB: local, zero cost, no limits.
• LlamaIndex: open-source, Apache 2.0.

Set your API key as an environment variable:

export GOOGLE_API_KEY="your-key-here"

Step 1: Scaffold the Project

Create a virtual environment and install dependencies. We're pinning versions that play nicely together as of early 2025.

mkdir codebase-qa && cd codebase-qa
python -m venv .venv && source .venv/bin/activate

pip install llama-index==0.11.10 llama-index-embeddings-gemini==0.2.0 \
  llama-index-llms-gemini==0.3.0 llama-index-vector-stores-chroma==0.2.0 \
  chromadb==0.5.5 tree-sitter==0.23.0 tree-sitter-languages==1.10.2 \
  pathspec==0.12.1

Wait—why tree-sitter? LlamaIndex's CodeSplitter uses it under the hood for AST-aware chunking. It understands function boundaries, class definitions, and scope. Without it, you'd be blindly splitting on newlines and hoping for the best. The pathspec library handles .gitignore pattern matching so we don't index node_modules or .venv.

Create the main script:

touch qa.py

Step 2: Ingest and Chunk the Repository

Open qa.py. We'll build the ingestion pipeline first. The key design decision: use CodeSplitter with language auto-detection so the chunker understands Python functions, JavaScript classes, Go structs, and so on.

import os
from pathlib import Path
from pathspec import PathSpec
from llama_index.core import SimpleDirectoryReader
from llama_index.core.node_parser import CodeSplitter
from llama_index.core.ingestion import IngestionPipeline

def load_gitignore(repo_path: str) -> PathSpec:
    """Parse .gitignore if it exists, otherwise return an empty spec."""
    gitignore_path = Path(repo_path) / ".gitignore"
    if gitignore_path.exists():
        with open(gitignore_path, "r") as f:
            return PathSpec.from_lines("gitwildmatch", f.readlines())
    return PathSpec.from_lines("gitwildmatch", [])

def file_filter(file_path: str, spec: PathSpec) -> bool:
    """Return True if the file should be indexed."""
    # Skip common binary and large-file extensions
    skip_extensions = {
        ".png", ".jpg", ".jpeg", ".gif", ".ico", ".pdf",
        ".zip", ".tar", ".gz", ".mp4", ".mov", ".lock", ".pyc"
    }
    if any(file_path.endswith(ext) for ext in skip_extensions):
        return False
    # Apply gitignore rules
    return not spec.match_file(file_path)

def ingest_repo(repo_path: str) -> list:
    spec = load_gitignore(repo_path)
    reader = SimpleDirectoryReader(
        input_dir=repo_path,
        recursive=True,
        exclude_hidden=False,  # We'll filter manually
        file_metadata=lambda fp: {"file_path": fp},
    )
    documents = reader.load_data()
    # Filter after loading so we have full paths
    filtered_docs = [
        doc for doc in documents
        if file_filter(doc.metadata["file_path"], spec)
    ]
    print(f"Loaded {len(filtered_docs)} files from {repo_path}")

    splitter = CodeSplitter(
        language="auto",
        chunk_lines=40,
        chunk_lines_overlap=15,
        max_chars=1500,
    )
    nodes = splitter.get_nodes_from_documents(filtered_docs)
    print(f"Created {len(nodes)} chunks")
    return nodes

chunk_lines=40 with 15 lines of overlap means each chunk captures roughly a function's worth of context. The overlap prevents splitting mid-function in a way that loses the signature. max_chars=1500 is a safety cap—Gemini's embedding model handles up to 3,072 tokens, but smaller chunks improve retrieval precision.

Step 3: Build the Vector Index with ChromaDB

Now we embed those chunks and persist them. LlamaIndex's IngestionPipeline handles batching and retries for the embedding API calls.

from llama_index.embeddings.gemini import GeminiEmbedding
from llama_index.vector_stores.chroma import ChromaVectorStore
from llama_index.core import VectorStoreIndex, StorageContext
import chromadb

def build_index(nodes: list, persist_dir: str = "./chroma_db"):
    embed_model = GeminiEmbedding(
        model_name="models/embedding-001",
        api_key=os.environ["GOOGLE_API_KEY"],
    )
    chroma_client = chromadb.PersistentClient(path=persist_dir)
    collection = chroma_client.get_or_create_collection("codebase")
    vector_store = ChromaVectorStore(chroma_collection=collection)
    storage_context = StorageContext.from_defaults(vector_store=vector_store)

    pipeline = IngestionPipeline(
        transformations=[embed_model],
        vector_store=vector_store,
    )
    pipeline.run(nodes=nodes, show_progress=True)

    index = VectorStoreIndex.from_vector_store(
        vector_store,
        embed_model=embed_model,
    )
    return index

models/embedding-001 is Google's latest embedding model as of early 2025. It produces 768-dimensional vectors. The free tier includes embedding requests in the same 1,500/day quota, so a codebase of 5,000 chunks costs 5,000 requests on first ingest. After that, ChromaDB serves queries with zero API calls.

Step 4: Wire Up the Gemini RAG Query Engine

With the index built, we create a query engine that retrieves relevant chunks and pipes them into Gemini for answer synthesis.

from llama_index.llms.gemini import Gemini

def create_query_engine(index: VectorStoreIndex):
    llm = Gemini(
        model="models/gemini-1.5-flash",
        api_key=os.environ["GOOGLE_API_KEY"],
        temperature=0.1,
    )
    query_engine = index.as_query_engine(
        llm=llm,
        similarity_top_k=5,
        response_mode="compact",
    )
    return query_engine

temperature=0.1 keeps answers deterministic—you want facts, not creative writing about your code. similarity_top_k=5 retrieves the five most relevant chunks. response_mode="compact" means LlamaIndex stuffs as many chunks as possible into the context window, then asks Gemini to synthesize. For gemini-1.5-flash with its 1M-token context window, this is rarely a bottleneck.

Step 5: The CLI Loop

Tie it together with an interactive loop that tracks usage and formats answers with source citations.

import sys

def main():
    if len(sys.argv) < 2:
        print("Usage: python qa.py <path-to-repo>")
        sys.exit(1)

    repo_path = sys.argv[1]
    persist_dir = "./chroma_db"

    # Ingest only if we haven't indexed this repo yet
    if not os.path.exists(persist_dir) or not os.listdir(persist_dir):
        print("First run: indexing repository...")
        nodes = ingest_repo(repo_path)
        index = build_index(nodes, persist_dir)
    else:
        print("Loading existing index...")
        embed_model = GeminiEmbedding(
            model_name="models/embedding-001",
            api_key=os.environ["GOOGLE_API_KEY"],
        )
        chroma_client = chromadb.PersistentClient(path=persist_dir)
        collection = chroma_client.get_collection("codebase")
        vector_store = ChromaVectorStore(chroma_collection=collection)
        index = VectorStoreIndex.from_vector_store(
            vector_store,
            embed_model=embed_model,
        )

    query_engine = create_query_engine(index)
    request_count = 0

    print("\nCodebase Q&A ready. Ask away (Ctrl+C to exit).\n")
    try:
        while True:
            query = input(">>> ").strip()
            if not query:
                continue
            if query.lower() in ("exit", "quit"):
                break

            response = query_engine.query(query)
            request_count += 1
            print(f"\n{response}\n")
            if response.source_nodes:
                print("Sources:")
                for node in response.source_nodes:
                    file_path = node.metadata.get("file_path", "unknown")
                    print(f"  - {file_path}")
                print()
            print(f"[Requests this session: {request_count}]\n")
    except KeyboardInterrupt:
        print(f"\nDone. Total Gemini API requests: {request_count}")

if __name__ == "__main__":
    main()

Running the Bot

Point it at any directory containing source code:

python qa.py ~/projects/my-express-api

First run indexes everything. Subsequent runs load the persisted ChromaDB index instantly. Try queries like:

• "Where is authentication middleware defined?"
• "What functions call the database connection pool?"
• "Explain the error handling pattern in the payment module."
• "Which files import the logger utility?"

The bot returns prose answers grounded in your code, with file paths so you can jump straight to the relevant source.

Sensible Extensions

Once the core loop works, here's where to take it next.

Watch mode for live repos: Wrap ingestion in a file watcher using watchdog. On file change, re-chunk only the modified file and upsert into ChromaDB. This keeps the index fresh without full re-indexing.

Multi-repo search: Extend the CLI to accept multiple repo paths. Prefix each document's metadata with a repo_name so answers can cite which project a file belongs to. ChromaDB collections handle this natively—create one collection per repo or use metadata filtering.

Slack bot integration: Wrap the query engine in a small HTTP server and connect it to a Slack slash command. If you've built something similar before, the pattern from our daily standup bot guide translates directly—swap the standup logic for a query_engine.query() call.

Email digest of codebase changes: Combine this with the approach in our email cold-outreach personalizer—schedule a weekly job that diffs the repo, summarizes changes with Gemini, and emails the team. Useful for large repos where PRs don't capture the full picture.

On-call incident helper: During an incident, point this at the relevant service repo and combine it with our incident summarizer guide. The summarizer handles logs and voice notes; the codebase bot answers "where is this error thrown?" in seconds.

Common Pitfalls and Fixes

"Rate limit exceeded" on first ingest: Large repos with thousands of files can blow through the 1,500 requests/day free quota. Solution: add a sleep(1) between embedding batches, or run ingestion overnight. The ChromaDB persistence means you only pay this cost once.

ChromaDB version mismatches: chromadb==0.5.5 and llama-index-vector-stores-chroma==0.2.0 are tested together. Newer ChromaDB versions change the client API. If you see AttributeError on get_or_create_collection, pin the versions exactly as shown.

Tree-sitter build failures: On macOS, pip install tree-sitter sometimes fails without Xcode CLI tools. Run xcode-select --install first. On Linux, you may need apt install build-essential.

Empty answers or hallucinations: If Gemini returns answers that don't reference your code, check that similarity_top_k isn't too low. Bump it to 10 or 15. Also verify your chunk_lines and chunk_lines_overlap aren't creating chunks too small to contain useful context.

Memory usage with large repos: ChromaDB loads the entire index into memory by default. For repos exceeding 100k chunks, switch to ChromaDB's client-server mode (still free, just run chroma run in a separate terminal) and connect via chromadb.HttpClient.

FAQ

Q: Does this work with private repos?
Yes. Everything runs locally except the embedding and generation calls to Google's API. Your source code never leaves your machine except as embeddings sent to Google. If that's a concern, you can swap in a local embedding model like all-MiniLM-L6-v2 via HuggingFace, though quality drops slightly.

Q: How do I re-index after a major repo refactor?
Delete the ./chroma_db directory and re-run. ChromaDB doesn't support incremental deletes easily, so a full rebuild is the cleanest path for large changes.

Q: Can I use OpenAI instead of Gemini?
Sure. Swap GeminiEmbedding for OpenAIEmbedding and Gemini for OpenAI. The architecture is identical. But OpenAI's free tier is more restrictive ($5 credit that expires), while Gemini's free tier is genuinely free for moderate usage.

Q: What if my repo has multiple languages?
CodeSplitter(language="auto") detects per-file. It uses file extensions to pick the right tree-sitter parser. Python, JavaScript, TypeScript, Go, Rust, Java, C++, and a dozen others work out of the box.

Q: How is this different from GitHub Copilot's codebase awareness?
Copilot indexes your repo for code completion. This bot is for question-answering and exploration. It's a different modality—you ask "why is this pattern used here?" rather than tab-completing a function. Both are useful; they complement each other.

Ready to go deeper? If you're integrating this into a production workflow with air-gapped environments, our case study on deploying LLMs behind strict firewalls covers the architectural patterns that keep things running when the internet isn't an option.