Deploying an LLM Feature at an Enterprise Customer: A Week-by-Week Case Study

Week 0: The Red-Eye and the Reality

The email hits at 4 PM on a Thursday. A Fortune 500 insurance customer has signed a six-figure deal, and the sales engineer’s demo—running on a public endpoint—won’t fly in their environment. They need a Forward Deployed Engineer on-site by Monday to make a document-summarization feature work behind their firewall. No pressure.

This isn’t a hackathon. This is the actual job: taking a promising, often fragile, LLM prototype and hardening it inside a customer’s specific, messy, and regulated infrastructure. The timeline is compressed, the stakes are high, and you are the human API between your company’s product and the customer’s reality.

Here’s exactly how a four-week enterprise LLM deployment unfolds, including the architecture, the code, the dead ends, and the career implications.

Week 1: Discovery in a SCIF

I land Sunday night. Monday morning, I’m badged into a Sensitive Compartmented Information Facility (SCIF)-like room where phones are locked in Faraday cages. The customer’s vision is clear: automatically summarize thousands of internal legal documents—think claims adjuster notes and policy documents—to accelerate settlement decisions.

The Real Discovery Questions

Forget the polished slide deck. The actual discovery happens at a whiteboard with the customer’s lead architect. The questions that matter:

1. The Network Boundary: “Can this server initiate an outbound connection at all, or is it a one-way diode?” (Answer: strict air-gap. No outbound internet. Period.)
2. The Data Shape: “Are these clean PDFs, or scanned TIFFs with hand-written margin notes?” (Answer: a mix of born-digital PDFs and 90s-era scanned documents.)
3. The Latency Budget: “Is this a synchronous API call during an adjuster’s workflow, or a nightly batch job?” (Answer: synchronous, and the internal SLA is 5 seconds end-to-end.)
4. The Human Factor: “Who is the ultimate adversary of this project, and what would make them kill it?” (Answer: the Chief Security Officer (CSO), who will block anything that transmits data off-premises.)

The Artifact: The One-Pager

I don’t leave the room without a one-page technical proposal. This document, co-authored with the customer’s architect, is my shield against scope creep. It specifies:

• Input: Base64-encoded PDF bytes.
• Output: A JSON object with a 3-sentence summary and 5 key entities extracted.
• Environment: RHEL 8, NVIDIA A100 80GB, Docker with no public registry access.
• Metrics: Success is 95% uptime and a human-evaluated summary accuracy score of 4/5.

This is the blueprint. If it’s not on this page in Week 1, it doesn’t ship in Week 4.

Week 2: The Air-Gapped Architecture

Back at my hotel (the only place with internet), I design the system. We can’t call OpenAI. We can’t call our own cloud. We have to bring a model in on a hardened drive and run it locally. This is the standard pattern for deploying LLMs in enterprise customer environments where data gravity and compliance are non-negotiable.

The System Flow

The architecture must be boringly reliable. No vector databases. No agentic loops. Just a straightforward inference pipeline.

The Stack

• Inference Engine: vLLM with a quantized Llama-3 70B (AWQ). Why? It saturates the A100’s memory bandwidth and gives us sub-second token generation on long contexts.
• OCR: Tesseract wrapped in a Python microservice. Not fancy, but it handles the 90s-era scanned docs without a GPU tax.
• API Layer: FastAPI. Synchronous, blocking calls are fine here because the customer’s workflow is a single-threaded desktop app.
• Containerization: Docker, but images are saved as .tar files and scanned by their security team before being loaded onto the air-gapped host.

The Code That Ships the Model

We can’t pip install on the machine. We have to bundle everything. A snippet from the Dockerfile that the security team will scrutinize:

FROM nvidia/cuda:12.4.0-runtime-ubuntu22.04

# Offline install: vLLM and dependencies pre-downloaded as wheels
COPY ./wheels /wheels
RUN pip install --no-index --find-links=/wheels /wheels/*.whl

# The model weights, scanned and approved
COPY ./models /models

COPY ./app /app
CMD ["python", "-m", "vllm.entrypoints.openai.api_server", \
     "--model", "/models/llama-3-70b-awq", \
     "--max-model-len", "8192"]

This is the critical FDE skill: translating a slick SaaS demo into a self-contained, auditable artifact that a security team can approve without asking you to change your core logic.

Week 3: Prompt Engineering Against a Brick Wall

With the model running, the real work begins. The customer’s legal documents are a nightmare of nested clauses, inconsistent terminology, and domain-specific shorthand that the base model hallucinates over.

The Failure Mode

Early prompts produced summaries that were fluent but factually inverted. “The claimant is liable” became “The claimant is not liable.” This is a trust-destroying bug in an insurance context. We can’t fix the model. We have to fix the prompt and the parsing.

The Prompt Strategy

We move to a constrained generation approach. Instead of asking for a free-text summary, we force the model to extract specific fields first, then synthesize. This is the core loop:

system_prompt = """
You are a legal document analyzer. Extract the following fields exactly as they appear.
Do not infer. Do not summarize. If a field is missing, return "NOT_FOUND".

Fields:
1. Claimant Name
2. Date of Loss
3. Policy Number
4. Liability Assessment (exact text)
"""

# ... after extraction, a second call with the extracted fields ...
summary_prompt = f"""
Based ONLY on the following extracted facts, write a 3-sentence summary.
Facts:
- Claimant: {claimant}
- Date: {date_of_loss}
- Liability: {liability}

Summary:
"""

The Guardrail

We implement a simple but effective hallucination detector: a string similarity check between the extracted Liability Assessment and the text in the summary. If the cosine similarity drops below 0.8, we flag it for human review and return a “Summary Unavailable – Manual Review Required” status. This kills the feature’s elegance but saves its credibility. The customer’s operations team loves it. They’d rather have a null than a lie.

Week 4: The Pilot and the Pivot

We go live with five senior adjusters. The first two days are quiet. Then the feedback arrives.

• The Good: “The entity extraction is saving me 15 minutes per file.”
• The Bad: “It’s choking on handwritten adjuster notes in the margins.”
• The Ugly: “The CSO wants an audit trail of every prompt and response for the next 90 days, stored in their SIEM.”

The Pivot

We can’t solve handwriting in 48 hours. We pivot: we add a preprocessing step that crops and saves margin regions as separate images, then returns them alongside the summary with a note: “Handwritten annotations detected. Please review.” This turns a failure into a feature.

For the audit trail, we implement a simple SQLite logger (no external database allowed) that captures the hash of the input document, the timestamp, the model version, and the full prompt/response pair. A cron job exports this to a CSV on a shared drive nightly, which their SIEM ingests. It’s not elegant, but it satisfies the compliance checkmark.

The Handoff

I spend the final two days writing the runbook. Not a README. A runbook: “If the GPU runs out of memory, run this script. If the OCR service crashes, restart it with this command.” I train two of their engineers, and I leave.

The feature processes 10,000 documents in the first month post-pilot. The customer expands the contract.

The FDE Career Context

This is why FDEs exist. A pure software engineer would have built a beautiful system that failed on the handwritten notes. A pure consultant would have made a slide deck about the failure. An FDE ships a working compromise that earns the right to iterate.

Compensation reflects this pressure. In 2025, an enterprise FDE with a track record of air-gapped deployments and LLM integration is commanding $200k–$280k base, with total comp reaching $350k+ when you factor in the performance bonuses tied to customer expansion revenue. The premium isn’t for coding ability; it’s for the judgment to know that a SQLite audit log and a “Manual Review” flag are the difference between a killed project and a renewed contract.

If you want to build this muscle, start by deploying a local LLM into a similarly constrained environment. A project like a SQL Analyst Agent that answers questions over a free Postgres database forces you to think about data that can’t leave the database, which is the same mental model as an air-gapped enterprise. The tools are open-source; the constraint is the teacher.

FAQ

What’s the hardest part of deploying an LLM in an enterprise? It’s rarely the model. It’s navigating the security review, building an audit trail that satisfies a CSO, and handling the dirty 10% of data (scanned PDFs, handwritten notes) that your demo never accounted for.

Do I need to be a machine learning engineer to do this? No. You need to be a strong backend engineer who understands how to serve a model via an API, craft a prompt, and parse the output. The FDE role values breadth and customer empathy over deep ML research.

How do you handle model updates in an air-gapped environment? You physically ship a new hardened drive with the updated weights and a new Docker image. The process is documented and repeated every quarter or as the contract dictates. It’s a logistics problem, not a software one.

What if the customer insists on using a model that doesn’t fit on their hardware? You show them the latency numbers on their own hardware. A 70B parameter model quantized to 4-bit that runs in 2 seconds is infinitely more valuable than a 180B model that doesn’t run at all. Data wins arguments in enterprise deployments.

Is on-site work always required? For the first deployment in a regulated industry, yes. Trust is built in person. For subsequent expansions, remote work is often possible once the integration pattern is established. The realities of on-site vs. remote FDE work are a balance of relationship-building and burnout management.