Deploying an LLM Feature at an Enterprise in 6 Days: An FDE Case Study

The Monday Morning Call: Scoping the Impossible

Monday 09:42. The Slack message from the Account Executive reads: "They need an internal knowledge assistant on their policy docs. Can’t use the cloud. Prod-ready by next Monday. Are we crazy?"

This is the standard Forward Deployed Engineer (FDE) opener. The customer—a global insurer—has 14,000 internal policy documents. Their claims adjusters waste 90 minutes a day searching a legacy SharePoint. The ask: a natural-language Q&A interface that runs entirely inside their VPC, never phoning home.

The constraints we signed up for:

• Environment: Bare-metal Kubernetes on-prem. No api.openai.com egress.
• Model: A frozen Mistral-7B-Instruct quantized to 4-bit. No fine-tuning allowed on their data yet.
• Timeline: 6 days to a working UAT deployment.
• Success metric: Answer accuracy on a held-out test set of 200 claims scenarios.

The FDE’s job isn’t just engineering—it’s collapsing the sales-to-value gap. At this point, the contract is signed, and the customer’s technical team is skeptical. We have one shot to prove the platform.

Architecture: The RAG Stack in a Suitcase

The pattern is Retrieval-Augmented Generation (RAG), but the enterprise constraints force specific choices. No vector databases requiring a separate cloud instance. No embedding APIs.

We settled on a self-contained architecture:

Why this stack:

• ChromaDB runs embedded in the Python process—no separate service, no Kubernetes operator. It’s not a petabyte-scale solution, but for 14K documents it’s fast and operationally invisible.
• BGE-small-en is a 384-dimensional embedding model that runs on CPU at 1000+ docs/sec. No GPU needed for ingestion.
• Mistral-7B-Instruct 4-bit fits in a single A10 GPU (24 GB VRAM) with room for KV cache. The customer had exactly two A10 nodes available.

Day 1-2: Data Ingress, Chunking, and the Vector Hurdle

The SharePoint export was a disaster. The customer’s IT team handed us a ZIP of 14,000 PDFs, half of which were scanned images with no OCR layer. Policies were split across multiple files with inconsistent headers. The first 6 hours were triage.

The Chunking Strategy That Saved the Project

Naive recursive character splitting failed immediately. A claims adjuster might ask, "What is the deductible for water damage under a commercial property policy?" The answer often spans two paragraphs separated by a table.

We implemented a document-structure-aware chunker:

def chunk_policy_document(text: str, metadata: dict) -> list[dict]:
    # 1. Split on section headers (regex for "Article IV", "Section 2.1", etc.)
    sections = re.split(r'(?=\n(?:Article|Section)\s+[IVX\d]+)', text)
    chunks = []
    for section in sections:
        # 2. If section > 1000 tokens, split on paragraph boundaries with 100-token overlap
        if token_count(section) > 1000:
            paragraphs = section.split('\n\n')
            current_chunk = ""
            for para in paragraphs:
                if token_count(current_chunk + para) > 900:
                    chunks.append({"text": current_chunk, "metadata": metadata})
                    # Overlap: keep last 2 paragraphs for continuity
                    current_chunk = "\n\n".join(current_chunk.split('\n\n')[-2:]) + "\n\n" + para
                else:
                    current_chunk += "\n\n" + para if current_chunk else para
            if current_chunk:
                chunks.append({"text": current_chunk, "metadata": metadata})
        else:
            chunks.append({"text": section, "metadata": metadata})
    return chunks

Metadata enrichment was critical. We extracted policy_type, effective_date, jurisdiction, and line_of_business from each document’s header. This metadata became filter criteria in ChromaDB, allowing us to scope retrieval to only relevant policies before semantic search.

The OCR problem: We deployed Tesseract in a sidecar container, but the quality was poor on handwritten endorsements. We flagged 200 documents for manual review and focused on the 13,800 machine-readable ones. This kind of pragmatic triage—knowing when to cut scope to hit the deadline—is core FDE skill.

By end of Day 2, we had 87,000 chunks embedded and indexed.

Day 3-4: Prompt Engineering Against a Static Model

With no fine-tuning allowed, the entire behavior of the system depended on the prompt template. Mistral-7B-Instruct is chat-tuned but not RLHF’d for insurance. It would hallucinate coverage limits, invent policy numbers, and occasionally refuse to answer with "I am not a legal expert."

The Retrieval Template

We iterated through 40+ prompt variants against our 200-question test set. The winning structure:

<s>[INST] You are a claims knowledge assistant for Acme Insurance.
You answer questions using ONLY the provided policy excerpts.
If the answer is not in the excerpts, say "I cannot find that information in the policy documents provided."
Never invent policy terms, numbers, or conditions.

Relevant policy excerpts:
---
{context}
---

Question: {question}

Answer (include the policy section reference when possible): [/INST]

Why this worked:

• Role anchoring: "Claims knowledge assistant" narrowed the model’s behavior space more than generic "helpful assistant."
• Explicit refusal path: Giving the model a precise phrase for out-of-scope questions reduced hallucination by 22% on our test set.
• Section reference requirement: Forcing citation grounded the answer in the retrieved text.

Retrieval Tuning

We started with top_k=4 chunks. This was too few for complex questions spanning multiple policy sections. top_k=10 introduced noise. We settled on top_k=6 with a similarity threshold of 0.65 and metadata pre-filtering.

The metadata filter was the unsung hero. For a question like "What is the flood deductible in Florida?" we first filtered to jurisdiction=FL and policy_type=property, then ran semantic search. This cut retrieval noise by 40%.

Day 5: The Guardrail Gauntlet and UX Injection

Enterprise legal and compliance teams don’t care about your chunking strategy. They care about three things:

1. Data leakage: Is PII exposed in responses?
2. Auditability: Can every answer be traced to a source document?
3. Jailbreaking: Can an adjuster trick the system into giving unvetted advice?

PII Redaction

We added a pre-processing step that ran Microsoft Presidio on all retrieved chunks before they entered the prompt. Any detected SSN, phone number, or name was replaced with [REDACTED]. This was a hard requirement from their InfoSec team.

The UX Hack That Closed the Deal

We embedded the answer inside a custom React component that showed the top 3 source chunks with similarity scores. Adjusters could click to open the original PDF at the cited section. This transparency turned skeptical claims managers into champions.

The component design was deliberately minimal—a single text input, a response card with a citation accordion, and a feedback thumbs-up/down. We instrumented the feedback to log to a local SQLite database for future fine-tuning data.

Day 6: UAT, Sign-Off, and the Silent Deployment

Friday morning: the claims VP tests 10 real scenarios from her team. The system correctly answers 8, partially answers 1, and misses 1 (a question about a rider that hadn’t been OCR’d). She says: "This is better than our current search. Ship it."

The silent deployment: We packaged everything into a single Helm chart. The FastAPI server, ChromaDB, and the React frontend all ran in the same pod. No external dependencies. The customer’s DevOps team deployed it to their staging namespace with a single helm install.

Metrics we tracked from day one:

The FDE Comp and Career Context

This 6-day sprint is not a hackathon project. It’s a paid engagement where the FDE is the technical linchpin of a 7-figure contract. The economics reflect that.

Comp bands for this type of work:

• Base salary: $160K–$220K for mid-to-senior FDEs at growth-stage companies. Palantir and similar primes push $190K–$250K base.
• Equity: 0.1%–0.5% at Series B-C companies; RSU packages at public cos.
• Variable/commission: Some FDE roles carry a 10–20% bonus tied to customer adoption metrics (not sales quotas).

For a deeper dive into the numbers and negotiation tactics, see Forward Deployed Engineer Compensation Bands and How to Negotiate Them.

The broader skill set—shipping AI features under enterprise constraints—is exactly what the market is pricing at a premium. As we’ve written before, AI Didn't Erase the Junior Engineer's Value—It Increased It. The FDE who can prompt-engineer a frozen 7B model into a compliant insurance assistant is not competing with the model; they’re orchestrating it.

This case study also connects to the deeper architectural principles in Designing Extensible Software in the Age of LLMs: Beyond Static APIs. The chunking pipeline and metadata filtering we built are not one-offs—they’re composable primitives that generalize across customers.

For a week-in-the-life view of how these sprints fit into the broader FDE rhythm, check out What a Forward Deployed Engineer Actually Does in a Week.

FAQ

Q: Why not use LangChain or LlamaIndex? A: We used neither. At the time, LangChain’s abstraction layers added debugging complexity that was unacceptable under a 6-day timeline. We wrote 300 lines of plain Python for retrieval and prompt assembly. The simplicity paid off when we had to debug a chunk boundary issue at 11 PM on Day 4.

Q: How did you handle model latency on CPU-only nodes? A: We didn’t. The customer had A10 GPUs. If they hadn’t, we would have used a smaller model (Phi-3-mini) or negotiated a batch-processing UX where answers returned in 30–60 seconds. CPU inference on a 7B model is not viable for interactive use.

Q: What happens after the 6-day sprint? A: The FDE typically hands off to a customer success engineering team with a runbook. In this case, I stayed embedded for 2 more weeks to collect feedback data, tune the prompt, and train the customer’s internal team on the chunking pipeline. The goal is always to make yourself replaceable.

Q: Is 87% accuracy good enough for production? A: It depends on the use case. For claims adjuster assistance—where a human always reviews the final decision—87% with source citation is transformative. For fully automated claims adjudication, it would be dangerously low. Setting the right success criteria is part of the FDE’s scoping job.