Beating GPT-4o on Retrieval with a 100x Cheaper Open Stack

The Claim: Open Models Beat GPT-4o on RAG

Neon, the serverless Postgres company, needed a retrieval pipeline for their documentation. The standard playbook says: throw everything at GPT-4o, pay the API tax, and move on. Instead, they built a fully open-source stack called Castform that outperforms GPT-4o on retrieval metrics—while costing roughly 100x less.

This isn’t a marginal win on a synthetic benchmark. They measured it on their own docs using real queries. The open pipeline scored higher on recall and answer quality, served responses faster, and eliminated the operational headache of rate limits and vendor lock-in.

The source post is worth reading in full: How Castform (Neon) beats frontier models on price and efficiency. What follows is the engineering autopsy—how it works, why it matters, and how you can build something similar this weekend.

The Castform Architecture: A Data-Engineer’s View

Most RAG pipelines follow a predictable shape: chunk documents, embed them, store vectors, retrieve at query time, then feed context to a frontier LLM. Castform keeps the skeleton but swaps every proprietary organ for an open-source one.

Here’s the flow:

Let’s walk through each component and why it was chosen.

1. Chunking Strategy

They use LangChain’s MarkdownHeaderTextSplitter. This isn’t arbitrary—Markdown-aware splitting preserves semantic boundaries (headers, code blocks, lists) that naive character splitters destroy. If you’ve ever debugged a RAG pipeline that returns half a function signature, you know this matters.

The splitter respects the document’s structural hierarchy: H2 sections stay together, code fences aren’t bisected, and list items remain grouped. This alone can improve retrieval precision by 10-20% on technical documentation.

2. Embedding Model: BGE-M3

BAAI’s BGE-M3 is the unsung hero here. It’s a multilingual embedding model that supports dense, sparse (lexical), and multi-vector retrieval in one model. For English-only tech docs, the dense vectors do the heavy lifting. But the model’s ability to handle 8192-token inputs means you can embed larger chunks without truncation loss.

Key specs:

• 1024-dimensional output vectors
• 8192 token context window
• Outperforms OpenAI’s text-embedding-3-large on MTEB retrieval benchmarks
• Runs on a single A10G or even CPU with quantization

They host it on Hugging Face Inference Endpoints, but you can run it locally with sentence-transformers or vLLM.

3. Vector Store: pgvector on Neon

Neon eats their own dog food. pgvector is a Postgres extension that adds vector similarity search. The advantage over Pinecone or Weaviate? Your vectors live alongside your operational data. No ETL to sync embeddings with metadata. No separate billing tier. Just SQL.

They use HNSW indexing with vector_cosine_ops for similarity. At their scale (thousands of docs), this is overkill—a flat index with exact search would work fine. But HNSW costs nothing to enable and future-proofs the pipeline.

4. Reranker: BGE-Reranker-v2-m3

This is where Castform pulls ahead of naive RAG. Retrieval grabs the top-k chunks (they use k=20), but the reranker—a cross-encoder that scores query-chunk pairs—reorders them by relevance. Only the top 3-5 make it to the LLM.

Cross-encoders are slower than bi-encoders (they process pairs, not individual texts), but they’re dramatically more accurate. BGE-Reranker-v2-m3 is state-of-the-art on the MTEB reranking leaderboard and runs comfortably on CPU for small batch sizes.

Why this matters: feeding irrelevant context to an LLM is worse than feeding no context. The reranker acts as a quality gate, ensuring the generator only sees high-signal information.

5. Generator: Llama 3.1 8B via Groq

Instead of GPT-4o, they use Meta’s Llama 3.1 8B Instruct, served through Groq’s LPU inference engine. Groq provides OpenAI-compatible endpoints with absurdly low latency (often sub-200ms for short generations).

The 8B model is small enough to be cheap but capable enough for grounded Q&A when given good context. The prompt template is straightforward: system message with retrieval context, user question, instruction to answer only from provided docs.

Why This Matters for Engineers and FDEs

If you’re a Forward Deployed Engineer or a product-minded engineer building customer-facing AI features, this architecture changes the economics of retrieval.

No More API Tax

GPT-4o charges $2.50/1M input tokens and $10/1M output tokens. For a documentation chatbot handling thousands of queries daily, that adds up fast. Llama 3.1 8B on Groq costs roughly $0.05/1M input and $0.08/1M output. That’s not a typo—it’s 50-100x cheaper.

Data Sovereignty

When you embed a customer’s proprietary documents, sending them to OpenAI’s API creates compliance headaches. An open stack can run entirely within a customer’s VPC or even on-prem. For FDEs embedding with enterprise clients, this is often the difference between a blocked POC and a signed deal. If you’re navigating that dynamic, the patterns in How Palantir-Style FDEs Embed with Customers to Unlock Technical Value apply directly.

Controllable Latency

OpenAI’s API latency varies with load. Groq’s LPUs deliver consistent sub-second responses. When you’re building interactive tools—like a Screenshot-to-React Agent with Google Gemini Flash—deterministic latency is non-negotiable.

Debuggability

When GPT-4o hallucinates, you file a support ticket. When Llama 3.1 8B hallucinates, you can inspect logits, modify the prompt, or fine-tune. The control surface is orders of magnitude larger.

The Cost Breakdown: 100x Isn’t Hyperbole

Let’s put numbers on it. Neon’s pipeline handles ~10,000 queries/month on their docs.

Neon reports their total monthly bill dropped from ~$400-500 to under $5. That’s two orders of magnitude.

For a single developer’s side project, this is the difference between “I’ll self-host” and “I can’t justify the cost.” For an enterprise deployment with millions of queries, it’s the difference between a six-figure annual line item and pocket change.

How to Replicate the Stack Today

You can build a Castform-style pipeline in an afternoon. Here’s the concrete path.

Step 1: Chunk Your Documents

from langchain_text_splitters import MarkdownHeaderTextSplitter

headers_to_split_on = [("##", "Header 2"), ("###", "Header 3")]
splitter = MarkdownHeaderTextSplitter(headers_to_split_on)
chunks = splitter.split_text(markdown_content)

If your docs aren’t in Markdown, convert them first. Pandoc handles most formats. The splitter respects header hierarchy, so nested sections stay together.

Step 2: Embed with BGE-M3

from sentence_transformers import SentenceTransformer

model = SentenceTransformer("BAAI/bge-m3")
embeddings = model.encode([chunk.page_content for chunk in chunks], normalize_embeddings=True)

For production, consider vLLM or Hugging Face TGI for higher throughput. The model fits comfortably on a T4 GPU.

Step 3: Store in pgvector

CREATE EXTENSION vector;
CREATE TABLE docs (id SERIAL PRIMARY KEY, content TEXT, embedding vector(1024));
CREATE INDEX ON docs USING hnsw (embedding vector_cosine_ops);

Neon’s free tier includes pgvector. If you’re already on Postgres, you’re three SQL statements away from a vector store.

Step 4: Add Reranking

from FlagEmbedding import FlagReranker

reranker = FlagReranker('BAAI/bge-reranker-v2-m3')
scores = reranker.compute_score([[query, chunk] for chunk in candidates])

Sort by score descending, keep top 3-5. The reranker adds ~100ms on CPU for 20 candidates.

Step 5: Generate with Llama 3.1 via Groq

import groq

client = groq.Groq(api_key="your_key")
response = client.chat.completions.create(
    model="llama-3.1-8b-instant",
    messages=[
        {"role": "system", "content": f"Answer using only this context:\n{context}"},
        {"role": "user", "content": query}
    ]
)

Groq’s free tier gives you enough tokens to prototype. For production, their paid tier is still orders of magnitude cheaper than OpenAI.

The Full Pipeline

If you want a pre-built starting point, this pattern mirrors the architecture in Build a Local RAG Chatbot Over Your PDFs with Ollama, LlamaIndex, and Qdrant Free Tier. Swap Qdrant for pgvector and OpenAI for Groq/Llama, and you’re 80% there.

For FDEs deploying this in customer environments, the handoff from prototype to production follows the maturity model described in Scaling Yourself: When an FDE Hands Off to Core Engineering for Productionization.

A Balanced Take: Where It Excels and Where It Doesn’t

Castform isn’t a universal GPT-4o replacement. It wins in a specific, well-defined context: retrieval-augmented generation over a bounded document corpus. Here’s the honest assessment.

Where It Shines

• Grounded Q&A over known docs. The combination of BGE-M3 embeddings, cross-encoder reranking, and a focused prompt makes hallucination rare.
• Cost-sensitive deployments. When query volume is high and margins are thin, 100x savings compound fast.
• Compliance-heavy environments. No data leaves your infrastructure. This matters for legal, medical, and financial use cases.
• Latency-critical applications. Groq’s LPUs are consistently fast. No cold starts, no queuing.

Where It Falls Short

• Complex reasoning. Llama 3.1 8B is not GPT-4o. For multi-hop questions, code generation, or nuanced analysis, the frontier model still wins.
• Broad knowledge. Without retrieval context, the 8B model’s world knowledge is limited. This pipeline is useless as a general chatbot.
• Multilingual edge cases. BGE-M3 handles 100+ languages, but generation quality in non-English languages lags behind GPT-4o.
• Maintenance overhead. You’re now responsible for model updates, endpoint health, and prompt engineering. For some teams, the OpenAI tax is worth the reduced ops burden.

The Takeaway

Castform proves that open models have crossed a critical threshold: for retrieval tasks, they’re not just “good enough”—they’re better. The key isn’t any single model; it’s the combination of a strong embedding model, a reranker as a quality gate, and a small but capable generator.

This stack is ideal for documentation chatbots, customer support assistants, and internal knowledge bases. It’s not a replacement for GPT-4o in open-ended reasoning tasks. But for the narrow, high-value use case of “answer questions from these documents,” it’s the engineering choice that makes sense.

If you’re building AI features that interface with customer systems, the patterns here pair naturally with the workflow in How FDEs Work with Product and Engineering Teams After the Enterprise Sale. The open-source stack gives you the flexibility to adapt to customer constraints without rearchitecting.

FAQ

Q: Can I run this entirely locally without any API calls? Yes. Replace Groq with Ollama running Llama 3.1 8B, use BGE-M3 locally via sentence-transformers, and run pgvector in a local Postgres instance. The entire stack fits on a machine with 16GB RAM and a consumer GPU.

Q: How does this compare to using LlamaIndex or LangChain’s built-in RAG? Those frameworks abstract away the component selection. Castform is a specific component recipe. You can implement it within LlamaIndex by swapping the default OpenAI embeddings and LLM for BGE-M3 and Llama 3.1. The reranker is the piece most default RAG pipelines miss.

Q: What’s the minimum hardware to run this in production? For low-volume (under 100 queries/hour): a 4GB GPU (T4 or consumer equivalent) for embeddings, CPU for reranking, and Groq’s API for generation. Total cost: under $50/month including hosting.

Q: Does the reranker really make that much difference? Yes. Without reranking, the top-5 chunks by vector similarity often include tangentially related content. A cross-encoder reranker improves answer accuracy by 15-30% on most retrieval benchmarks. It’s the highest-ROI component you can add to a RAG pipeline.

Q: Can I swap in a different open model for generation? Absolutely. Mistral 7B, Qwen 2.5 7B, and Phi-3 Medium all work well. The architecture is model-agnostic. Choose based on your latency budget and quality requirements.