Case Study: Deploying an LLM Feature at an Enterprise Customer in 10 Days

The Brief: A Spreadsheet and a Skeptical VP

A $2B industrial manufacturer signed a six-figure platform deal. The ink was dry, but the VP of Engineering, Mark, didn’t believe the “AI” module would work on his data. His challenge: “You have 10 days. My team has spent 6 months trying to get a useful internal search running. If you can’t beat it, we shelf the AI SKU.”

The goal wasn’t a demo. It was a production-shippable feature that ingested 15,000 unstructured engineering Failure Mode and Effects Analysis (FMEA) reports and answered natural-language questions like: “What is the most common failure mode for the Series-9 actuator under thermal load?”

This is the case study of how a Forward Deployed Engineer (FDE) shipped that feature in 10 days, including the architecture, the code, the comp impact, and the brutal enterprise constraints.

Day 0: The 3-Hour Triage

Enterprise triage isn’t about code. It’s about finding the actual blockers before you write a line. I spent the first three hours in a windowless conference room with Mark and his lead architect, Sarah.

The Constraints:

1. Air-gapped VPC: No outbound internet from the data store. No OpenAI API.
2. PDF Hell: The 15,000 FMEA reports were split across SharePoint folders, a legacy Documentum system, and a shared drive. Half were scanned images.
3. Audit Trail: Every LLM response had to be traceable to a source document paragraph for compliance.

The Decision: We couldn’t use a managed cloud LLM. We needed a self-hosted open-weight model. We chose Llama 3 70B quantized to 4-bit (GGUF format) running on an on-prem NVIDIA A100 the customer already owned. For the embedding model, we used BGE-Large-en-v1.5 for retrieval.

The Comp Reality: This is why FDEs command $180k–$300k+ TC. You aren’t just writing fetch() calls. You are making architectural bets in a high-pressure, high-ambiguity environment where the technical decision directly determines whether a six-figure renewal happens.

Architecture: The “No Egress” Constraint

The architecture had to be 100% on-prem, with no external API calls. Here’s the flow we designed:

Why this stack:

• vLLM: Served the quantized Llama 3 70B with continuous batching. We got ~25 tokens/sec, acceptable for an internal tool.
• pgvector: Sarah’s team already managed a Postgres instance. Adding the pgvector extension was a 5-minute change request instead of a 3-week security review for a new vector database.
• HyDE (Hypothetical Document Embeddings): Critical for this domain. Users queried “actuator failure,” but documents said “torque degradation due to thermal expansion.” HyDE generates a hypothetical document from the query, embeds that, and uses it for retrieval. It bridges the vocabulary gap without fine-tuning.

Day 1–3: The “Trojan Horse” Data Connector

The biggest risk wasn’t the LLM. It was the data extraction. Scanned PDFs in engineering often have rotated text, multi-column layouts, and technical diagrams.

The Stack:

• Unstructured.io (open-source): We deployed it inside their VPC. It handled PDFs, images (OCR via Tesseract), and even embedded Excel tables.
• Custom chunking strategy: We didn’t chunk by token count. We chunked by document section using the FMEA report structure (e.g., “Failure Mode,” “Effect,” “Cause,” “Mitigation”). This preserved semantic boundaries.

# Simplified: Custom section-aware chunker for FMEA reports
import re
from unstructured.partition.pdf import partition_pdf

def chunk_fmea_report(file_path: str):
    elements = partition_pdf(file_path, strategy="hi_res")
    full_text = "\n".join([el.text for el in elements])
    
    # Split by FMEA section headers
    sections = re.split(r'(?=Failure Mode:|Effect\(s\):|Potential Cause\(s\):|Recommended Action:)', full_text)
    chunks = []
    for section in sections:
        if len(section.strip()) > 100:
            chunks.append({
                "text": section.strip(),
                "metadata": {"source": file_path}
            })
    return chunks

The Result: By Day 3, we had 85,000 clean chunks in pgvector. Mark’s team had spent months on this step. The difference wasn’t AI magic—it was using the right tool (Unstructured) and domain-specific chunking.

Day 4–6: Evaluation-Driven Prompt Engineering

We didn’t guess at prompts. We built a 50-question evaluation set with Sarah’s engineers. Each question had a gold-standard answer and the expected source document.

The Evaluation Loop:

1. Run retrieval on the 50 questions.
2. Measure recall@5 (did the gold document appear in the top 5 chunks?).
3. For the LLM response, measure faithfulness (did the answer hallucinate facts not in the context?) using a simple LLM-as-judge script.

The Prompt That Won: After 40 iterations, here’s the system prompt that maximized faithfulness:

You are an engineering assistant for [Company Name]. Answer ONLY using the provided context below. 
For every claim, cite the source document and paragraph number in brackets [Doc: <filename>, Para: <number>]. 
If the context does not contain the answer, state "Insufficient data in the provided FMEA reports." 
Do not use outside knowledge.

The citation requirement wasn’t just for compliance. It acted as a constraint that reduced hallucination by forcing the model to ground every statement.

Key Metric: Faithfulness went from 72% to 96%. Recall@5 hit 91% after we tuned HyDE’s generation prompt to use engineering terminology.

Day 7: The “Aha!” Moment (UX Hack)

Sarah’s engineers tested the system and complained: “It’s accurate, but I don’t want to type a question. I want to look at a report and ask ‘what else fails like this?’”

This is the FDE superpower: hearing a complaint and shipping a feature in hours, not sprints.

The “Similar Failures” Button: We added an endpoint that took the current report’s chunk embedding and ran a vector similarity search across all other reports. The result was a “Similar Failure Modes” panel that appeared when viewing any report.

# FastAPI endpoint for similar failure modes
@router.post("/similar")
async def similar_failures(request: SimilarRequest):
    # Embed the input text
    embedding = await embed_text(request.text)
    
    # Cosine similarity search in pgvector
    results = await db.fetch(
        """SELECT text, metadata, 1 - (embedding <=> $1) AS similarity 
           FROM fmea_chunks 
           WHERE 1 - (embedding <=> $1) > 0.85 
           ORDER BY similarity DESC LIMIT 5""",
        embedding
    )
    return results

This feature took 4 hours to build and deploy. It became the most-used feature in the entire application. Mark later told me this was the moment he knew he’d renew.

Day 8–10: Hardening and the “Scream Test”

The last three days weren’t about new features. They were about making sure the system didn’t fail silently.

What we did:

• Guardrails: We implemented a simple output validator that checked if citations actually existed in the provided context. If a hallucination was detected, the system returned a fallback: “I could not find sufficient evidence in the reports.”
• Monitoring: We instrumented the vLLM server with Prometheus metrics (TTFT, throughput, queue depth) and built a Grafana dashboard.
• The “Scream Test”: On Day 10, we let 20 engineers loose on the system with no guidance. We watched silently. Within 30 minutes, they’d found the “Similar Failures” feature and were using it to cross-reference reports. Zero crashes.

The Renewal: Mark signed the AI SKU expansion. The six-figure deal became a seven-figure annual contract.

Comp & Career Context

This case study isn’t just about tech. It’s about the economic reality of FDE work.

The Math:

• FDE TC: $220k (base + bonus + equity).
• Contract Value Saved/Expanded: $800k ARR.
• Value Ratio: 3.6x annual comp in 10 days.

This is why FDEs are compensated differently than pure software engineers. You are measured on customer outcomes, not story points. When you can point to a case study like this in a performance review or interview, you aren’t asking for a raise—you’re presenting a business case.

For more on how to position this kind of work in your career, read What an FDE Actually Does in a Week and The FDE Interview Loop: How to Prepare for Execution, Not LeetCode Crimes.

FAQ

Q: Why not just use OpenAI’s API with an Azure private endpoint? A: The customer’s security policy explicitly prohibited any data leaving their physical data center, even via a private link to a cloud provider. The A100 was already on-site and underutilized.

Q: How did you handle the scanned image PDFs? A: Unstructured.io’s hi_res strategy uses a combination of Tesseract OCR and layout detection models. We had to install Tesseract and the English language pack on the processing server, but it handled 98% of the scanned reports without issue.

Q: What was the hardest technical challenge? A: Tuning HyDE’s generation prompt. The hypothetical document it generated had to use domain-specific terminology (e.g., “torque degradation” instead of “breaking”). We solved this by including 3 example FMEA report excerpts in the HyDE prompt as few-shot examples.

Q: Was this a one-off or is this pattern repeatable? A: The pattern—self-hosted open-weight model, pgvector, Unstructured.io, HyDE retrieval—is highly repeatable for any enterprise with unstructured document Q&A needs and air-gapped constraints. The custom chunking strategy and evaluation set are the parts that require domain adaptation.

Q: How do I learn to do this kind of work? A: The best preparation is building end-to-end projects that span data ingestion, retrieval, and LLM serving under constraints. For a hands-on example of a similar RAG deployment pattern, see the Deploy a Natural Language SQL Analyst Agent walkthrough. For more on the enterprise embedding playbook, read How Palantir-Style FDEs Embed with Customers.