Case Study: Deploying an LLM Feature Behind an Enterprise Firewall in 2 Weeks

The 2-Week Constraint and Customer Context

A Fortune 500 insurance carrier needed an internal policy Q&A tool. The ask was deceptively simple: upload a 2,000-page claims adjustment manual and let adjusters ask natural language questions. The catch? The manual contained proprietary actuarial tables and claimant PII patterns. Legal mandated zero data egress to public cloud LLM APIs. The entire system had to run inside their Azure tenant, behind their firewall, with their existing SSO.

They had already tried an internal hackathon build using a naive chunk-and-embed approach on a local GPU box. It failed spectacularly on multi-hop reasoning questions like "If a claimant has a pre-existing condition in a no-fault state, what is the maximum reserve I can set without supervisor approval?" The answers were plausible but wrong—the most dangerous failure mode in insurance.

As the Forward Deployed Engineer on the account, I had two weeks to ship a working pilot that was accurate, auditable, and deployable entirely within their boundary. This is the playbook for how we did it.

Architecture: Keeping Data In-Region and Off-Internet

The non-negotiable constraint was that no prompt text, document chunk, or embedding vector could leave the customer's Azure East US region. This ruled out OpenAI, Anthropic, Cohere, and every managed vector database. We had to assemble the stack from self-hosted components that could run in their VPC.

The flow is standard RAG at first glance: chunk documents, embed them, store vectors, retrieve at query time, stuff into prompt, generate. But every component had to be selected for self-hosted compatibility and the customer's existing infrastructure. No SaaS dependencies.

The Technical Stack: Why We Chose What We Chose

Model: Llama 3 70B Instruct (AWQ quantized) via vLLM We needed strong instruction following and a permissive license. Llama 3 70B was the best open-weight model at the time that could run on a single A100 node. The AWQ 4-bit quantization let us fit it in 40GB of VRAM with acceptable quality loss. vLLM gave us continuous batching and a clean OpenAI-compatible API, which made the FastAPI layer trivial.

Embeddings: all-MiniLM-L6-v2 via Sentence Transformers We didn't need a massive embedding model for ~2,000 pages of text. The MiniLM model runs on CPU, produces 384-dim vectors, and is fast enough for real-time retrieval. Crucially, it runs entirely locally with no network calls.

Vector Store: Qdrant We evaluated Milvus, Weaviate, and pgvector. Qdrant won because it's a single Rust binary with no external dependencies, has excellent filtering (we needed metadata filtering by policy line and effective date), and the customer's ops team was comfortable managing it. We deployed it on a Standard_D4s_v3 with a managed disk.

Document Parsing: Unstructured.io The claims manual was a mix of PDFs with complex tables, scanned appendices, and Word docs. Unstructured's library handled table extraction and layout parsing better than PyPDF2 or pdfplumber. We ran it as a preprocessing step, outputting clean markdown chunks with metadata.

The Build Sprint: Day-by-Day Breakdown

Days 1-2: Infrastructure and Procurement The biggest risk was hardware. The customer's Azure tenant had GPU quota for NCas_T4_v3 but not for A100s. We filed an emergency quota increase request, but the approval SLA was 5 business days. The workaround: we found a spare NC96ads_A100_v4 in their dev subscription that a data science team wasn't using. We commandeered it with a Monday morning email to their VP. Always map the org chart before you need it.

Days 3-5: Data Pipeline and Chunking Strategy The naive chunking from the hackathon was the root cause of the bad answers. They had split on 512-token boundaries with no overlap. When a policy rule referenced a table on the next page, the retriever couldn't pull both chunks.

We rebuilt the pipeline with three changes:

1. Table-aware parsing: Unstructured.io extracted tables as structured markdown, preserving row/column relationships.
2. Semantic chunking: We used a sentence splitter with 256-token chunks and 64-token overlap, but only split on section boundaries detected by header patterns.
3. Metadata enrichment: Every chunk got tagged with policy_line, effective_date, jurisdiction, and a parent section_id.

# Example: metadata-enriched chunking with Unstructured
from unstructured.partition.auto import partition
from unstructured.chunking.title import chunk_by_title

elements = partition(filename="claims_manual.pdf", strategy="hi_res")
chunks = chunk_by_title(
    elements,
    max_characters=1500,
    overlap=200,
    multipage_sections=True
)

for chunk in chunks:
    chunk.metadata.update({
        "policy_line": extract_policy_line(chunk),
        "effective_date": extract_date(chunk),
        "source_page": chunk.metadata.page_number
    })

Days 6-8: Retrieval Tuning and Prompt Engineering With the chunks in Qdrant, we ran a battery of 50 test questions the customer's SMEs provided. Initial retrieval precision@5 was 0.62—not good enough. We added:

• Hybrid search: Dense (vector) + sparse (BM25 via Qdrant's keyword index). The BM25 component caught exact matches on policy codes like "CL-402(b)(iii)" that embeddings missed.
• Multi-hop retrieval: For questions with "if...then...what" patterns, we did two retrieval passes. The first pass pulled the conditional rule. The second pass used the rule's output to query for the consequence.
• Re-ranking: We used a cross-encoder (ms-marco-MiniLM-L-6-v2) to re-rank the top-20 retrieved chunks down to the top-5 that went into the prompt.

The prompt template itself was deliberately structured to force citation:

You are a claims adjustment policy assistant. Answer using ONLY the provided context.
If the context doesn't contain the answer, say "I cannot find this in the policy manual."

Context:
{chunks}

Question: {query}

Answer (with citations to policy section and page):

Days 9-10: Enterprise Auth and Deployment The customer used Azure AD with SAML. We integrated the FastAPI app with MSAL for Python, validating JWT tokens on every request. The frontend was a simple Streamlit app (they already had Streamlit in their approved software catalog) that called the FastAPI backend.

Days 11-12: Red Teaming and Edge Cases We spent two days with the customer's SMEs trying to break the system. The most valuable exercise was adversarial: they asked questions designed to trick the model into hallucinating PII or revealing policy gaps. We caught several failures where the model would confabulate a plausible-sounding policy section number. Adding the explicit "cannot find" instruction reduced these by ~80%.

Days 13-14: Documentation and Handoff The FDE deliverable isn't just working code—it's an operations runbook. We documented the VM restart procedure, the Qdrant snapshot schedule, the model update process, and a troubleshooting guide for common failure modes.

The Hardest Part: Enterprise Auth and Data Connectors

Nobody talks about this in LLM tutorials. The model and vector store were the easy part. The hard part was:

1. SharePoint connector: Half the policy documents lived in SharePoint, not Azure Blob. We had to write a custom connector using the Microsoft Graph API with delegated permissions, navigating a maze of tenant admin consent.
2. PII scrubbing: The claims manual contained real example data with names and claim numbers. We built a regex + spaCy NER pipeline to redact these before chunking, but false positives (redacting policy codes that looked like claim numbers) required manual review.
3. Audit logging: Legal required every query and response to be logged immutably for 7 years. We piped everything to Azure Log Analytics with a retention lock.

These are the things that separate a demo from a production deployment—and they're where FDEs earn their comp.

When It Goes Wrong: Debugging Hallucinations on Proprietary Data

On Day 11, a tester asked: "What is the statute of limitations for filing a supplemental claim in Louisiana after a hurricane?" The model confidently answered "2 years from the date of loss." The correct answer was 1 year per Louisiana Revised Statute 22:868. The model had retrieved the general claims filing deadline (2 years) but missed the hurricane-specific carve-out buried in a footnote.

The fix wasn't a better model. It was better chunking. The hurricane exception was in a footnote on page 847, separated from the main text by a page break. Our chunker had split it into its own chunk with no connection to the parent section. We added a post-processing step that detected footnote references and merged footnote chunks back into their parent section chunks.

This is the core FDE skill: not tuning hyperparameters, but understanding the data and the domain well enough to know why the system is failing in the specific ways that matter to the customer.

Results and What We'd Do Differently

After two weeks, we had a system that:

• Achieved 89% accuracy on the SME test set (up from 54% on the hackathon version)
• Served responses in 3-8 seconds (acceptable for internal tooling)
• Ran entirely within the customer's Azure tenant with zero egress
• Passed the legal and infosec review

The adjusters adopted it immediately. The average time to answer a policy question dropped from 12 minutes (searching the PDF manually) to under 30 seconds.

What we'd do differently:

1. Start with the evaluation set on Day 1. We spent 3 days building the pipeline before we had a ground truth test set. Build the eval first, then optimize against it.
2. Push harder on the GPU quota. The commandeered A100 was a single point of failure. We should have escalated the quota request to the customer's Azure account team on Day 0.
3. Consider a smaller model. Llama 3 8B with 4-bit quantization might have been sufficient for this single-domain task, and would have run on a cheaper VM. We over-provisioned.

FAQ: Enterprise LLM Deployment

What are some LLM use cases in the enterprise setting? The most common we see: internal knowledge base Q&A (like this case), contract analysis and clause extraction, claims/policy triage, RFP response generation, and code migration assistants. The common thread is high-volume text processing where the domain is proprietary and accuracy matters more than creativity. For more on building these kinds of internal tools, see our guide on building a daily standup bot that collects updates via DM.

What is a key challenge of deploying LLMs in customer service? The hallucination-auditability tradeoff. In customer-facing settings, a wrong answer has legal and reputational risk. The mitigation is a combination of constrained retrieval (RAG with strict citation), human-in-the-loop for high-stakes answers, and prompt engineering that forces the model to express uncertainty. The same patterns apply to agentic workflows—check out our piece on Docker sandboxes for AI agents for isolation strategies.

How does SAP make LLMs relevant and reliable for enterprise? SAP's approach (via their Joule copilot and BTP integration) focuses on grounding LLMs in structured business data from S/4HANA and SuccessFactors, not just unstructured text. They use a semantic layer that maps natural language to business objects with strict authorization checks. This is the same principle we applied with metadata filtering in Qdrant—the LLM shouldn't see data the user isn't authorized to access.

What is an enterprise LLM? An enterprise LLM is a large language model deployed within an organization's security boundary, integrated with its identity provider, data stores, and audit systems. It's less about the model architecture and more about the operational wrapper: access controls, data residency, monitoring, and the ability to update or roll back model versions without downtime. For FDEs, this is the core job—making frontier AI work inside the constraints of real companies. If you're building your skills in this area, our self-study curriculum for FDEs covers the infrastructure and integration patterns that matter.

Can I run a useful LLM entirely on-premises without a GPU cluster? Yes, for many single-domain use cases. Quantized 7-8B parameter models run on consumer GPUs or even high-end CPUs with acceptable latency for internal tools. The bottleneck is usually not the model size but the quality of retrieval and the domain-specific chunking strategy. For an extreme example of local deployment, see how Muse Glimmer runs a 30B agent model entirely on a laptop.