Build a Local Codebase Q&A Tool with Ollama, LlamaIndex & ChromaDB

What We're Building

A command-line tool that points at any local Git repository, parses every source file, embeds the code with a free local LLM, stores it in a vector database, and lets you ask natural-language questions. Think: "Where is authentication logic implemented?" or "What functions touch the database connection pool?"

Feature list:

• Recursive ingestion of a repo (respects .gitignore)
• Chunking that preserves file boundaries and function-level context
• Embedding generation via Ollama + DeepSeek Coder V2 (free, runs on your machine)
• Local vector storage with ChromaDB (persistent, zero cost)
• Retrieval-augmented generation (RAG) query loop
• Fully offline—no API keys, no rate limits, no telemetry

If you’ve built the job-application autofill agent before, you’ll recognize the local-first pattern. This time we’re indexing code, not web forms.

Architecture Overview

Here’s how the pieces fit together:

The ingestion path (left column) runs once. The query path (right column) runs per question. Both rely on Ollama serving two models: an embedding model for vectorization and DeepSeek Coder V2 for generation.

Prerequisites

Everything is free and runs locally.

After installing Ollama, pull both models:

ollama pull deepseek-coder-v2:16b
ollama pull nomic-embed-text

Verify they appear:

ollama list

If you’re low on RAM, swap the 16B model for deepseek-coder-v2:6.7b—still free, still local, just lighter.

Step 1: Scaffold the Project

Create a directory and a virtual environment:

mkdir codebase-qa
cd codebase-qa
python -m venv .venv
source .venv/bin/activate  # Windows: .venv\Scripts\activate

Install dependencies:

pip install llama-index llama-index-embeddings-ollama llama-index-llms-ollama chromadb

Create three files:

codebase-qa/
├── ingest.py      # One-shot indexing
├── query.py       # Interactive Q&A
└── config.py      # Shared settings

Step 2: Ingest and Chunk the Codebase

We need a reader that handles .py, .js, .ts, .go, .rs, and other source files. SimpleDirectoryReader can filter by extension. For chunking, LlamaIndex’s CodeSplitter respects language-specific AST boundaries when available, falling back to character splits.

config.py

import os
from pathlib import Path

# Point this at your repo
REPO_PATH = os.environ.get("REPO_PATH", str(Path.home() / "my-project"))

# ChromaDB persistence directory
CHROMA_PATH = os.environ.get("CHROMA_PATH", "./chroma_db")

# Collection name
COLLECTION_NAME = "codebase"

# Embedding model (must be pulled in Ollama)
EMBED_MODEL = "nomic-embed-text"

# LLM for generation
LLM_MODEL = "deepseek-coder-v2:16b"

# File extensions to index
ALLOWED_EXTENSIONS = [".py", ".js", ".ts", ".tsx", ".go", ".rs", ".java", ".c", ".cpp", ".h", ".rb"]

ingest.py

import os
from pathlib import Path
from llama_index.core import (
    SimpleDirectoryReader,
    VectorStoreIndex,
    StorageContext,
    Settings,
)
from llama_index.core.node_parser import CodeSplitter
from llama_index.embeddings.ollama import OllamaEmbedding
from llama_index.vector_stores.chroma import ChromaVectorStore
import chromadb

import config

def main():
    # 1. Set up embedding model globally
    Settings.embed_model = OllamaEmbedding(
        model_name=config.EMBED_MODEL,
        base_url="http://localhost:11434",
    )

    # 2. Read source files
    reader = SimpleDirectoryReader(
        input_dir=config.REPO_PATH,
        required_exts=config.ALLOWED_EXTENSIONS,
        recursive=True,
        exclude_hidden=True,
    )
    documents = reader.load_data()
    print(f"Loaded {len(documents)} documents")

    # 3. Chunk with CodeSplitter (falls back to text split for unsupported languages)
    splitter = CodeSplitter(
        language="python",  # default; LlamaIndex auto-detects per file where possible
        chunk_lines=40,
        chunk_lines_overlap=15,
        max_chars=1500,
    )
    nodes = splitter.get_nodes_from_documents(documents)
    print(f"Created {len(nodes)} nodes")

    # 4. Initialize ChromaDB and create index
    chroma_client = chromadb.PersistentClient(path=config.CHROMA_PATH)
    chroma_collection = chroma_client.get_or_create_collection(config.COLLECTION_NAME)
    vector_store = ChromaVectorStore(chroma_collection=chroma_collection)
    storage_context = StorageContext.from_defaults(vector_store=vector_store)

    index = VectorStoreIndex(
        nodes,
        storage_context=storage_context,
        embed_model=Settings.embed_model,
    )
    print(f"Index built. Vectors stored in {config.CHROMA_PATH}")

if __name__ == "__main__":
    main()

What’s happening:

• SimpleDirectoryReader walks the repo, skipping hidden files and respecting the extension whitelist.
• CodeSplitter tries to split on function/class boundaries for Python and a few other languages. For unsupported languages, it falls back to character-window splitting.
• ChromaDB persists vectors to disk so you only index once.

Step 3: Embed and Store in ChromaDB

Run ingestion:

ollama serve  # if not already running
python ingest.py

Expected output:

Loaded 342 documents
Created 2147 nodes
Index built. Vectors stored in ./chroma_db

ChromaDB writes a SQLite database plus Parquet files into ./chroma_db. You can inspect it with:

import chromadb
client = chromadb.PersistentClient(path="./chroma_db")
collection = client.get_collection("codebase")
print(collection.count())  # number of vectors

Time depends on repo size. A 50 MB codebase takes ~2-3 minutes on a MacBook M1. The heavy lift is embedding generation—Ollama runs the model on CPU or GPU depending on your setup.

Step 4: Build the Q&A Query Pipeline

Now the fun part: retrieval-augmented generation. We load the persisted index, wire up DeepSeek Coder V2 as the LLM, and build a query engine.

query.py

import chromadb
from llama_index.core import (
    VectorStoreIndex,
    Settings,
    StorageContext,
)
from llama_index.core.query_engine import RetrieverQueryEngine
from llama_index.embeddings.ollama import OllamaEmbedding
from llama_index.llms.ollama import Ollama
from llama_index.vector_stores.chroma import ChromaVectorStore

import config

def build_query_engine():
    # Embedding model (same as ingestion)
    Settings.embed_model = OllamaEmbedding(
        model_name=config.EMBED_MODEL,
        base_url="http://localhost:11434",
    )

    # LLM for generation
    Settings.llm = Ollama(
        model=config.LLM_MODEL,
        base_url="http://localhost:11434",
        request_timeout=120.0,
        temperature=0.1,  # low temp for factual code answers
    )

    # Load existing ChromaDB collection
    chroma_client = chromadb.PersistentClient(path=config.CHROMA_PATH)
    chroma_collection = chroma_client.get_collection(config.COLLECTION_NAME)
    vector_store = ChromaVectorStore(chroma_collection=chroma_collection)

    index = VectorStoreIndex.from_vector_store(
        vector_store,
        embed_model=Settings.embed_model,
    )

    # Configure retriever
    retriever = index.as_retriever(similarity_top_k=8)

    # Build query engine with a custom prompt template
    from llama_index.core.prompts import PromptTemplate

    qa_prompt = PromptTemplate(
        "You are an expert software engineer analyzing a codebase.\n"
        "Answer the question using ONLY the code context provided below.\n"
        "If the context doesn't contain the answer, say 'I couldn't find that in the codebase.'\n"
        "Include file paths and function names when relevant.\n"
        "\n"
        "Context:\n"
        "{context_str}\n"
        "\n"
        "Question: {query_str}\n"
        "Answer:"
    )

    query_engine = RetrieverQueryEngine.from_args(
        retriever=retriever,
        llm=Settings.llm,
        text_qa_template=qa_prompt,
    )

    return query_engine

def main():
    query_engine = build_query_engine()
    print("Codebase Q&A ready. Type 'exit' to quit.\n")

    while True:
        question = input("Ask > ").strip()
        if question.lower() in ("exit", "quit"):
            break
        if not question:
            continue

        response = query_engine.query(question)
        print(f"\n{response}\n")

if __name__ == "__main__":
    main()

Key design decisions:

• temperature=0.1 keeps answers grounded—no hallucinated code.
• similarity_top_k=8 retrieves enough context for cross-file questions without blowing the context window.
• The prompt explicitly constrains the model to the retrieved context, which dramatically reduces confabulation.

Step 5: Run It From the CLI

python query.py

Example session:

Ask > How does authentication work?

The authentication logic is implemented in `auth/middleware.py` in the
`AuthMiddleware` class. It validates JWT tokens using the `verify_token`
function from `auth/jwt_utils.py`. Session management happens in
`auth/session.py` via the `SessionManager` class.

Ask > What functions call `get_db_connection`?

`get_db_connection` is called in:
- `db/repository.py`: `UserRepository.get_user()`
- `db/repository.py`: `UserRepository.create_user()`
- `api/handlers.py`: `handle_login()`

If you get vague answers, increase similarity_top_k to 12-15. If answers are slow, switch to the 6.7B model.

Sensible Extensions

1. GitHub repo cloning Add a --repo-url flag that clones the repo to a temp directory before ingestion:

import subprocess, tempfile
# clone, run ingest, clean up

2. Watch mode for active development Use watchdog to detect file changes and re-index only modified files. This keeps the vector store in sync without full re-ingestion.

3. Web UI with Streamlit Wrap the query engine in a Streamlit app (free, open-source). 20 lines of code gets you a browser interface. This pattern echoes what we covered in the customer-review sentiment dashboard—same principle, different domain.

4. Multi-repo search Maintain separate ChromaDB collections per repo and route queries based on user selection.

5. Metadata filtering Tag nodes with language or directory during ingestion, then filter retrieval:

retriever = index.as_retriever(
    similarity_top_k=8,
    filters={"language": "python"},
)

This is especially powerful in monorepos. For a deep dive on structured retrieval, the GigaToken tokenizer deep-dive shows how token-level optimizations cascade into retrieval performance.

Common Pitfalls and Debugging

“Ollama connection refused” Ensure ollama serve is running. Check http://localhost:11434 in your browser—you should see “Ollama is running.”

“Model not found” You forgot to ollama pull. Run ollama list to confirm both models are present.

ChromaDB “collection already exists” errors on re-ingestion Delete the collection before re-ingesting:

chroma_client.delete_collection("codebase")

Or increment the collection name in config.py.

Answers reference non-existent files The LLM is hallucinating. Lower temperature (try 0.0), increase similarity_top_k, or tighten the prompt to forbid invention.

Out-of-memory during embedding The embedding model batches documents. If you hit OOM, reduce batch size by setting embed_batch_size=8 on the OllamaEmbedding constructor.

Slow responses DeepSeek Coder V2 16B is a chunky model. If you’re on a machine without a GPU, expect 5-10 seconds per query. The 6.7B variant cuts that to 2-4 seconds with a small accuracy trade-off. This is the same reasoning-effort vs cost trade-off that applies to all LLM pipelines.

FAQ

Q: Does this work on Windows? Yes. Use PowerShell, activate the venv with .venv\Scripts\activate, and ensure Ollama is in your PATH.

Q: Can I index a private repo? Absolutely. Everything runs locally. No code leaves your machine.

Q: How large a repo can this handle? I’ve tested up to ~500 MB (10k+ files). Beyond that, consider sharding ChromaDB or filtering to specific directories. Embedding time scales roughly linearly with total character count.

Q: What if I want to use OpenAI embeddings instead? Swap OllamaEmbedding for OpenAIEmbedding. You’ll need an API key and internet access. The free local stack is the point here, but the architecture supports any embedding provider LlamaIndex wraps.

Q: How do I get better answers for complex architectural questions? Two levers: (1) chunk granularity—smaller chunks give more precise retrieval but lose context; (2) the prompt template—add explicit instructions like “trace the call graph” or “list all files involved.” Experiment with both.

Q: Where do I go from here? This tool is a foundation. The FDE customer prototype playbook shows how tools like this ship in a week when a customer asks, “Can you make our 10-year-old codebase searchable?” If you’re building these skills for a forward-deployed role, FDE Coach has the playbooks, compensation data, and real-world patterns to accelerate you.