Case Study: Deploying an LLM Feature at an Enterprise Customer in 3 Weeks

It’s Monday morning. You’re an FDE and a customer success manager is pinging you with a “quick question” that is neither quick nor a question. A $2B logistics company has 15,000 pages of internal SOPs scattered across SharePoint. They want a “ChatGPT for our dispatchers”—but it has to run inside their VPC, never touch the public internet, and be live for a pilot in three weeks.

This isn’t a thought experiment. It’s a real case study of deploying an LLM feature at an enterprise customer. No benchmarks, no Kaggle datasets—just an air-gapped environment, a skeptical security team, and a deadline that doesn’t move.

The Engagement: A $2B Logistics Nightmare

The customer’s problem was deceptively simple: dispatchers spent 40% of their day searching SOPs to answer edge-case questions like, “Can a refrigerated trailer carrying pharmaceuticals be reassigned mid-route if the temperature log shows a 15-minute deviation?” The answer existed in a PDF. Somewhere.

Their IT team had already tried a keyword-search tool. It failed because dispatchers don’t search with keywords—they ask messy, context-heavy questions. They needed retrieval-augmented generation (RAG), but every SaaS solution was dead on arrival because the data couldn’t leave their network.

As an FDE, your job isn’t to sell them a product. It’s to ship a working feature that proves the value, then figure out how to make it stick. This is a core pattern from the FDE week-in-life: you’re doing discovery, engineering, and stakeholder management simultaneously.

Week 1: Scoping, Security, and the VPC Gauntlet

Day 1-2: The Art of Ruthless Scoping

We locked the scope to one document repository (the pharmaceutical cold-chain SOPs) and three question types. This wasn’t a product—it was a feature. The goal was to prove that an LLM could answer with citations, not to build a universal search engine.

I wrote a one-page spec that looked like this:

• Input: Natural language question from a dispatcher
• Output: A concise answer with verbatim source quotes and document names
• Constraints: Runs on a single VM with no outbound connections; latency under 5 seconds

This spec became my shield. Any “what if we also…” request got deflected with, “Great idea for Phase 2. Let’s ship Phase 1 first.”

Day 3-4: The VPC Gauntlet

The security review was the real bottleneck. Their architecture requirements were:

• Model inference must run entirely on-premises inside their VPC
• No telemetry, logging, or network calls to external services
• All data processing stays within their subnet

This ruled out OpenAI, Anthropic, and any hosted embedding service. The solution was to run everything locally using open-weight models. For the embedding model, we used all-MiniLM-L6-v2 via SentenceTransformers. For the LLM, we deployed Llama 3 8B quantized to 4-bit via llama.cpp, which runs comfortably on a single A10 GPU or even a high-memory CPU instance if latency requirements are relaxed.

The architecture flow looked like this:

This is the kind of pragmatic architecture FDEs live in. You’re not designing for millions of users—you’re designing for 50 dispatchers who need an answer in under 5 seconds. The stack is intentionally boring: FastAPI, ChromaDB in ephemeral mode, and llama.cpp with a Python binding. No Kubernetes, no microservices. A single docker-compose.yml that runs on their VM.

Week 2: The RAG Architecture (No Vector DB Hype)

Ingestion: Turning PDFs into Trustworthy Chunks

The SOPs were a mess—scanned PDFs with handwritten margin notes, mixed orientations, and tables that broke standard parsers. I spent two days writing a preprocessing pipeline:

1. OCR correction using Tesseract for the scanned pages
2. Table extraction with Camelot for tabular SOPs (temperature ranges, drug names, handling codes)
3. Chunking with a custom splitter that respected section boundaries—never splitting a numbered procedure mid-step

Each chunk was stored with metadata: source document name, page number, and section heading. This metadata is what makes citations work. When the LLM responds with “According to SOP-PHARMA-042, Section 3.2…”, the dispatcher can click and verify.

Retrieval: Why Cosine Similarity Won (and BM25 Lost)

I ran a quick eval on 20 real dispatcher questions. BM25 (keyword search) failed on 14 of them because the questions used synonyms that didn’t appear in the docs. Dense retrieval with all-MiniLM-L6-v2 got 18 out of 20 right. The two failures were questions requiring multi-hop reasoning—which I solved by adding a second retrieval step triggered by specific patterns in the question.

This is where the FDE skill of building fast, pragmatic evals pays off. You don’t need a ground-truth dataset of 10,000 Q&A pairs. You need 20 questions that cover the edge cases your champion at the customer gave you.

Week 3: Prompt Engineering, Evals, and the Demo

The Prompt That Shipped

After 40+ iterations, here’s the system prompt that stuck:

You are a logistics SOP assistant. Answer the question using ONLY the provided context.
If the context doesn't contain the answer, say "I cannot find this in the SOPs."
Always cite the source document name and section number.

Context:
{chunks}

Question: {question}
Answer:

Three things matter here:

• “Using ONLY the provided context” prevents hallucination when the answer isn’t in the chunks
• The explicit fallback (“I cannot find this…”) trains the user that silence is better than fabrication—a lesson reinforced by the SQLite CVE hallucination incident
• Citation formatting is baked into the prompt, not post-processed

The Eval Framework

I built a 30-question test set with the customer’s SME. Each question had an expected answer and a list of required citations. The eval script ran all 30 questions, checked for citation presence, and flagged any hallucinated document names. This took two hours to build and saved the project.

During testing, the model confidently cited “SOP-TRANS-089” on a question about pharmaceutical transport. That document didn’t exist. The model had invented it. Adding a strict “cite only document names present in the context” constraint to the prompt fixed it.

The Demo

The demo wasn’t a slide deck. It was a live terminal session. I asked the dispatcher team lead to type her hardest question. She typed: “If a reefer unit fails in transit and the cargo is insulin, what’s the maximum time before the shipment is non-conforming per FDA guidelines?”

The system returned in 3.2 seconds:

Per SOP-PHARMA-017, Section 4.2: “Insulin shipments experiencing a temperature excursion above 8°C must be quarantined if the duration exceeds 120 minutes.” FDA guideline 21 CFR 211.94 is referenced as the governing standard.

She turned to her VP and said, “This would have saved me three hours last Tuesday.” That’s the moment the project went from “interesting prototype” to “we’re funding this.”

The Handoff: From Prototype to Production

The three-week sprint proved the concept. But an FDE’s job isn’t done when the demo works—it’s done when the customer can run it without you. This is the FDE-to-core-engineering handoff pattern.

What I handed off:

• A docker-compose.yml that ran the entire stack
• A 10-page runbook covering model updates, chunk re-ingestion, and common failure modes
• The 30-question eval set with a CI script they could run after any change
• A one-pager for their security team documenting exactly which files touched the network (none, after the initial model download)

What I didn’t hand off: a production system. That’s their core engineering team’s job. My prototype proved the architecture was viable; they’ll add monitoring, auth, and scale. This delineation is critical—FDEs who try to build production systems burn out and create maintenance nightmares.

Comp & Career Context

This kind of engagement is why FDE roles command $180K–$250K+ at top AI companies. You’re not just writing code—you’re navigating enterprise security, managing skeptical stakeholders, and making architectural decisions under tight constraints. The skills compound: the prompt engineering techniques from this logistics project directly apply to the next healthcare engagement, and the VPC deployment pattern becomes a reusable template.

If you’re building toward this role, the portfolio piece that gets attention isn’t a side project with a flashy UI—it’s a case study like this one, showing you can ship under real constraints. Build something concrete that solves a specific, messy problem.

---

FAQ: Enterprise LLM Deployment

What’s the hardest part of deploying LLMs in enterprise VPCs? Not the model. The security review. Plan for a week of back-and-forth on data flow diagrams, dependency audits, and proving that your embeddings model isn’t phoning home. Start the security conversation on Day 1.

How do you choose between open-weight and API-based models? If the data can’t leave the network, the decision is made for you. Llama 3, Mistral, and Qwen models quantized with llama.cpp are the standard toolkit. For local dev workflows, Qwen3.8-Max coding benchmarks show these models are surprisingly capable.

What’s the minimum hardware for a usable RAG pipeline? A single VM with 32GB RAM and an A10 GPU can run a 7-8B parameter model quantized to 4-bit, plus the embedding model, with sub-5-second latency for typical document Q&A. For smaller models or CPU-only, techniques like layer-wise loading can squeeze 70B models onto surprisingly constrained hardware.

How do you prevent hallucination in enterprise settings? Three layers: (1) a prompt that forbids answering without context, (2) citation verification in your eval pipeline, and (3) a user expectation that “I don’t know” is an acceptable answer. The SQLite CVE hallucination case is required reading here.

Is three weeks actually realistic? Yes, if you scope ruthlessly. One document type. Three question types. One VM. The FDE superpower is saying “no” to scope creep while making the customer feel heard about Phase 2. The FDE toolkit is built for exactly this speed.