Case Study: Deploying a RAG-Powered LLM Feature at a Regulated Enterprise

The Deployment Environment: Air-Gapped and Audited

Three weeks into the engagement, I was standing in a windowless SOC-II compliant room staring at a bare-metal HPE ProLiant server that had never touched the public internet. The customer—a top-20 U.S. property and casualty insurer—had a single requirement that shaped every subsequent decision: no claim data, policy language, or personally identifiable information (PII) could leave their virtual private cloud.

This is the reality of deploying an LLM feature at a regulated enterprise. It’s not about picking the shiniest model on the OpenRouter leaderboard. It’s about making a 7B-parameter open-source model useful enough that an adjuster with 20 years of experience trusts it to summarize a 400-page commercial liability policy.

Forward Deployed Engineers (FDEs) exist precisely at this friction point. The role isn't pure research, and it isn't traditional solutions architecture. You are the person who writes the Python connector that talks to a legacy Guidewire claims system on a Tuesday, then presents the latency p95 metrics to the CISO on a Thursday. This case study walks through the actual workflow of deploying a Retrieval-Augmented Generation (RAG) feature in that environment, including the comp structure that makes it worth the travel.

Data Engineering: Chunking Strategy for Policy Documents

The insurer’s problem was straightforward to articulate but brutal to execute: new claims adjusters spent 45-60 minutes per complex claim manually searching across five different policy administration systems to determine coverage. The goal was a co-pilot that ingested structured policy PDFs and rendered a cited, accurate coverage summary in under 30 seconds.

We evaluated three chunking strategies against a golden dataset of 50 policies annotated by senior adjusters:

We built a custom pypdf parser that extracted the document tree based on font sizes and indentation—not just text coordinates. Insurance policies have a deep hierarchical structure (Title > Chapter > Section > Clause). Preserving that parent-child relationship in the metadata payload was the single highest-leverage decision for retrieval accuracy.

The FDE lesson: Enterprise documents are not flat text files. You must reverse-engineer the document object model (DOM) of their specific format. The code block below shows the metadata structure we attached to every vector embedding:

# Metadata payload per chunk stored in ChromaDB
metadata = {
    "policy_id": "POL-2024-09234",
    "section_title": "Exclusions - Water Damage",
    "hierarchy": ["Part IV", "Liability Exclusions", "Section C"],
    "page_number": 87,
    "effective_date": "2024-01-01",
    "line_of_business": "Commercial Property"
}

Architecture: On-Prem RAG Without Data Exfiltration

Since no data could leave the VPC, we couldn’t use managed embedding APIs. The architecture had to be fully self-contained. The diagram below outlines the final deployment topology.

We selected llama-3-8b-instruct quantized to 8-bit running on vLLM. Why 8B? It was the largest model we could reliably serve with sub-2-second time-to-first-token on a single A100 80GB, which was the only GPU tier the infrastructure team had approved for the data center. Embeddings were generated with BAAI/bge-large-en-v1.5, running locally via HuggingFace Transformers. The vector store was ChromaDB in persistent mode, backed by a local SSD volume.

The critical integration was the ETL connector to Guidewire. Guidewire’s on-prem instance exposed a set of SOAP APIs (yes, in 2024). We built a thin translation layer in FastAPI that converted the claim context—injured party details, date of loss, policy number—into a structured JSON payload that seeded the RAG retrieval query.

The Security Review: Prompt Injection and RBAC

Enterprise security teams don’t care about your BLEU score. They care about two things: data exfiltration vectors and role-based access controls (RBAC).

We passed the security review by demonstrating three specific guardrails:

1. Indirect Prompt Injection Mitigation: We didn’t just strip user input; we used a separate, tiny classifier model (distilbert-base-uncased) fine-tuned on a custom dataset of 500 adversarial insurance prompts to detect "jailbreak" and "ignore previous instructions" patterns before the text hit the LLM. If the classifier score exceeded a threshold, the request was blocked and logged.
2. RBAC at the Document Level: An adjuster in the "Workers’ Comp" department could not query policies from the "Commercial Auto" line. We enforced this by injecting a WHERE line_of_business = $user_dept filter into the ChromaDB metadata query. The LLM never saw documents it shouldn’t.
3. Immutable Audit Trail: Every prompt, retrieved context chunk, and final generated response was hashed (SHA-256) and written to a separate audit log database before rendering to the user. This allowed the compliance team to replay any interaction and prove the model didn’t hallucinate a coverage amount—it misread a specific source document, which is a very different regulatory liability.

Operationalizing the Model: System Prompts and Guardrails

Getting retrieval accuracy to 0.91 was necessary but not sufficient. The model still occasionally exhibited sycophancy (agreeing with an adjuster’s incorrect assumption) or verbosity (generating legal advice). We solved this with a strict system prompt, heavily influenced by techniques detailed in our piece on Claude System Prompts: Operationalizing Model Behavior at the API Layer.

The prompt enforced a rigid output schema:

You are a coverage analysis assistant. You strictly adhere to the source documents.

RULES:
1. Answer ONLY using the provided context.
2. If the context lacks the answer, state "Insufficient policy data to determine coverage."
3. You MUST cite the specific policy section (e.g., "Per Section IV.C.2...") for every statement.
4. Do NOT offer legal advice or interpretations. Summarize the policy language verbatim.
5. Never suggest the adjuster is wrong. State the facts from the policy.

Context:
{retrieved_chunks}

Query: {user_question}

We further constrained the output using lm-format-enforcer, which guaranteed valid JSON output on structured fields, preventing the model from drifting into free-text narratives that compliance would flag.

Comp and Career Context for FDEs

Why does an FDE do this work instead of a resident solutions architect? Speed and ownership. As discussed in How AI-Native Startups Use FDEs to Win and Expand Enterprise Deals, the FDE carries a quota or a direct revenue influence target. My comp plan for this engagement was split 70/30 base/variable, with the variable portion tied directly to this feature passing the insurer’s User Acceptance Testing (UAT) gate within 6 weeks.

That variable upside—often $50k-$80k annually for senior ICs—is the premium paid for tolerating the travel reality described in On-Site vs Remote FDE Work: Travel Realities, Burnout, and Comp Implications. I spent 12 days on-site that month, sleeping in a hotel adjacent to the data center. The technical toolkit I carried is detailed in The Tools an FDE Ships With: Data Pipelines, Integration Scaffolds, and Demo Kits.

The hard truth: deploying LLMs in the enterprise is 20% model selection and 80% data engineering, security compliance, and ruthless scope management. The FDE role compensates for this difficulty with high autonomy and direct client influence. If you enjoy the puzzle of making a good-enough model work perfectly within Byzantine constraints, it’s the best engineering job in the market.

FAQ

What is the biggest technical risk when deploying RAG in a regulated industry? Hallucination with false citations. The model can invent a plausible-sounding policy section that doesn’t exist. Mitigation requires schema-enforced generation and a strict system prompt that prohibits the model from inventing section numbers not present in the retrieved context.

How do you handle model updates in an air-gapped environment? We delivered quantized model weights (GGUF format) on encrypted physical USB drives via bonded courier. The hash was verified against a public ledger before loading into vLLM. This process was documented in the customer’s Model Risk Management (MRM) framework.

Why not use a larger, more capable model? Infrastructure approval cycles in large enterprises move slowly. The A100 server was already procured and racked. An FDE optimizes for the constraints that exist, not the constraints they wish existed. The 8B model, combined with precise chunk retrieval, surpassed the accuracy requirements. Larger models offered diminishing returns relative to the latency cost.

Is this role just "consulting"? No. Consultants hand over a slide deck. FDEs hand over a running codebase with 95% test coverage on the ETL connectors and a runbook for the MLOps team. The distinction is shipping production artifacts against a revenue target.