Case Study: Deploying an LLM Feature at an Enterprise Customer as an FDE

The Call: A 'Simple' Document Problem

It started with a Slack message from the Account Executive: "Global insurance client. They have 400k unstructured medical reports in PDFs. They need to extract 12 specific fields into their claims system. They tried a generic SaaS solution and it failed on their messy tables. They have a $1.2M ACV expansion on the line. Can you fly out Monday?"

This is the standard opening move for a Forward Deployed Engineer (FDE). You aren't just handed a Jira ticket. You are handed a business-critical problem, a frustrated customer, and a ticking clock. The enterprise isn't buying a model; they are buying a business outcome. Shipping that outcome requires engineering chops, but also a specific type of customer archaeology that defines the role.

Discovery: The Art of the First On-Site

I landed Monday morning and went straight to their claims processing center. I didn’t open my laptop for the first two hours. Instead, I sat with Maria, a senior claims adjuster, and watched her work.

This is the highest-leverage activity an FDE can do: process mapping. Maria’s screen showed the horror of enterprise UI: a green-screen AS/400 emulator, a modern web-based claims dashboard, and a folder of PDFs. Her workflow was to manually read the PDF, find the "Diagnosis Code," the "Date of Loss," and the "Attending Physician," and re-type them into the green screen. It took 8 minutes per report. She processed 40 a day.

The previous SaaS AI tool failed because it treated the PDF as a flat text stream. Maria showed me a specific report where the diagnosis was inside a complex bordered table that spanned two pages. The text extraction lost the tabular context entirely. The LLM didn't fail; the ingestion pipeline did.

The Technical Discovery: I took 20 sample PDFs back to the "war room" (a small conference room they gave us). I ran a quick Python script locally to classify the PDF types. We weren't dealing with clean digital text. We were dealing with:

1. Scanned images (no embedded text layer).
2. Hybrid PDFs (text layer that was corrupted/out of order).
3. Tabular data that broke standard bounding-box parsing.

This immediately killed the idea of a simple textract -> prompt pipeline. We needed a vision model, not just a text model. This is a classic FDE pivot: the customer says they want "AI extraction," but the technical reality is they need "computer vision for document understanding."

Architecture: Why We Killed the Vector DB

The client’s CTO joined the afternoon session. He had a white paper from a big consulting firm recommending a RAG (Retrieval-Augmented Generation) architecture with a vector database for "grounding." He wanted to know how we were going to chunk the documents.

I had to push back. Politely. We weren't doing semantic search over a knowledge base. We were doing structured extraction on a single document at a time. A vector DB added latency, cost, and hallucination risk by retrieving irrelevant chunks of other medical reports.

Our architecture decision flow:

We designed a zero-shot extraction pipeline. We didn't fine-tune a model. We didn't train a custom OCR. We used a frontier vision model (Gemini 1.5 Pro) with an extremely detailed prompt that included the exact schema and examples of the messy tables.

Why this worked: The model’s context window allowed us to feed in the entire high-resolution page image. For multi-page tables, we stitched the images vertically. The model could "see" the table borders and understand the relational context that text-based extraction missed.

The Security Gauntlet: VPC, IAM, and Data Residency

This is where most demos die. The client’s security team had a strict rule: No PHI (Protected Health Information) leaves the VPC. Our standard cloud API endpoint was a non-starter.

This is the defining technical challenge of enterprise FDE work. You are not just a prompt engineer; you are an infrastructure engineer.

The Solution: Private Cloud Run + VPC Service Controls We deployed a containerized microservice on Google Cloud Run inside their VPC (they were a GCP shop, but the pattern is identical on AWS with ECS/Fargate). The key components:

1. Ingress: Internal TCP Load Balancer with Private Service Connect.
2. Egress: We had to allow traffic to the Vertex AI API endpoint, but only through a VPC Service Controls perimeter that ensured data stayed within the us-east4 region.
3. IAM: We used workload identity federation. No long-lived API keys. The service account had exactly one permission: aiplatform.endpoints.predict on the specific model endpoint.

# Simplified auth pattern in the service
# No keys stored in env vars
credentials = google.auth.default()
aiplatform.init(project='client-project', location='us-east4')
model = aiplatform.GenerativeModel('gemini-1.5-pro-001')
response = model.generate_content([image_part, prompt])

We also had to implement a data retention policy in code. The service was stateless. It processed the image in memory, streamed the JSON response to the claims system, and wiped the memory buffer. No logs contained PHI. This took 48 hours of pairing with their security architect, but it was the only way to get the green light.

Shipping the Prototype: 48 Hours to 'Wow'

The "Friday Demo" is an FDE tradition. You don't have weeks. You have until the end of the trip to show something that makes the budget holder lean forward.

I built a thin Streamlit frontend (not production-grade, but visually clear) that showed a side-by-side of the original PDF and the extracted JSON. We ran it against Maria's 20 "nightmare" documents.

The first run extracted 11 of 12 fields correctly. The "Date of Loss" failed on one document because the date format was "Nov 3rd, 2023" and our Pydantic validator expected YYYY-MM-DD.

The fix: We didn't adjust the prompt. We adjusted the validation layer. We added a pre-processing step in the Pydantic validator that used dateutil.parser to normalize any date string. The model was returning the right information; our rigid schema was rejecting it. This is a critical FDE lesson: validate loosely, parse strictly.

Maria watched the demo. She didn't say "Wow, that's a great model." She said, "Wait, I can clear my queue by 2 PM?" That's the moment you know you've won.

Production Hardening: Prompts, Guardrails, and Fallbacks

Moving from the demo to production took another three weeks of remote work. The core challenges:

1. Prompt Drift: We version-controlled our prompts in a Git repository, separate from the application code. This allowed the client’s medical experts to suggest prompt tweaks via pull requests without touching the Python code. We used a YAML file:

# prompts/extraction.yaml
system: "You are a medical document parser. Extract the fields exactly as they appear..."
schema:
  - name: diagnosis_code
    type: string
    pattern: "^[A-Z][0-9]{2}"
  - name: date_of_loss
    type: date

2. Guardrails: We implemented a "semantic outlier" detector. If the model extracted a diagnosis code that had never appeared in the historical database, we flagged it for human review. This wasn’t about model accuracy; it was about catching edge cases like new ICD-11 codes that the model hadn't seen.

3. Fallbacks: If the Vertex AI API returned a 429 (rate limit), we implemented exponential backoff. If it returned a 5xx, we failed over to a secondary model endpoint in a different region. This is the unglamorous reality of FDE work: you write a lot of retry logic.

The FDE Compensation and Career Context

This deployment unlocked a $1.2M expansion. The FDE’s role in that is direct and measurable. You are not a cost center; you are visibly attached to revenue.

This visibility translates directly into compensation. While a backend engineer at a product company might optimize a query that saves $10k a month, an FDE can point to a specific logo they retained or a specific expansion they unlocked. For a deeper dive into how this translates into numbers, see our breakdown of FDE Compensation Bands and How to Negotiate: Equity, Base, and Sign-On.

This case study also highlights the week-to-week reality of the role. You aren't just coding. You are doing security architecture, sitting with users, and writing retry logic. It’s a blend that requires a specific temperament. If you’re curious what a typical calendar looks like, read What a Forward Deployed Engineer Actually Does in a Week: Code, Customers, Chaos.

If you’re coming from a pure backend or frontend role and wondering how to build these muscles, we have a roadmap for that transition in How to Break Into FDE Roles from a Backend or Frontend Background.

FAQ

Q: Do I need to be a machine learning expert to be an FDE? No. You need to be a strong software engineer who understands how to integrate ML models. You need to know enough about prompting, tokens, and context windows to make the model sing, but you don't need to know how to write a backpropagation algorithm. The hardest part of this case study was the VPC networking and the Pydantic validation, not the model architecture.

Q: What if the customer insists on a bad architecture (like the vector DB)? You push back with data. Build a quick benchmark. Show the latency and accuracy difference. But ultimately, if they insist, you build it. Part of the job is navigating enterprise politics. You can’t win every battle on day one.

Q: How do you handle the travel? It varies. Some FDE roles are 50% travel, some are 10%. The key is to make the on-site visits count. You don't travel to "check in." You travel for discovery, demos, and firefights. The rest you handle remotely.

Q: Is this just consulting? No. Consultants typically hand off a slide deck or a prototype and leave. An FDE writes production code that lives inside the customer’s environment, often for years. You carry a pager for the thing you built. You own the outcome.