Case Study: Deploying an LLM Feature That Survived Enterprise Security Review

The Setup: $2M on the Line, Zero Trust

It was a Thursday morning when the Slack message hit: "Legal killed the PoC. LLM data egress risk." We were three weeks into a pilot with a top-10 US bank. The feature was simple—an AI copilot that summarized internal policy documents for relationship managers. The contract was $2M ARR if we went live. The problem: the bank’s security team had just classified all generative AI as "unacceptable risk" until proven otherwise.

This is the reality of deploying an LLM feature at an enterprise customer. It’s not about the model’s BLEU score. It’s about surviving a security review that treats your inference endpoint like a data exfiltration device. As the Forward Deployed Engineer (FDE) on the account, I owned the technical win. Here’s exactly how we navigated the gauntlet, the architecture we landed on, and the career leverage you gain when you become the person who can ship AI past InfoSec.

Architecture: The Local-First, Zero-Egress Pattern

The bank’s policy documents were sensitive. Internal strategy memos, risk assessments, HR guidelines. Sending them to an external API (even Azure OpenAI with “no training” clauses) was a hard no from their CISO. We needed to run inference where the data lived, without phoning home.

We landed on an architecture pattern I now call Local-First, Zero-Egress. The principle: the LLM runs inside the customer’s VPC (or on-prem data center), and no prompt or completion ever leaves that boundary. For this deployment, we used an open-source model (Llama 3 70B) served via vLLM on their internal GPU cluster. Our application code sat in a sidecar container, handling retrieval-augmented generation (RAG) against their internal document store.

Here’s the flow we designed to make the security team comfortable:

The critical design choice: the vLLM server had no network egress rules. We configured Kubernetes NetworkPolicies to deny all outbound traffic from the inference pod. The sidecar handled all external interactions (like pulling updated document embeddings) through a strict allowlist. This gave the security team a clean boundary to audit.

The Security Gauntlet: 5 Questions That Stopped the Room

Every enterprise LLM deployment hits the same wall: the security review questionnaire. Here are the five questions that nearly killed our project, and the exact answers that got us through.

That last point—the immutable log—was the linchpin. We built a lightweight sidecar that intercepted the prompt/response stream, hashed the content with SHA-256, and shipped only the hashes to their SIEM. This let them prove no data leaked without ever storing sensitive text in a monitoring tool.

The Deployment: Air-Gapped Containers and Signed Artifacts

Getting the model into their environment was its own adventure. The bank’s artifact registry didn’t allow pulling from Docker Hub or Hugging Face directly. We had to pre-package everything into signed, air-gapped containers.

Our build pipeline looked like this:

1. Export Model Weights: Download Llama 3 70B from a trusted source, verify SHA256 checksum.
2. Bake Into Image: Create a vLLM Docker image with the weights baked in (not mounted as a volume—this was a security requirement for immutability).
3. Sign with Cosign: Use Sigstore’s Cosign to sign the image with our private key. The bank’s admission controller verified this signature before allowing the pod to schedule.
4. Deliver via Physical Media (Yes, Really): Their most sensitive environment was completely disconnected. We shipped the signed container on an encrypted drive via courier. The FDE life isn’t always glamorous.

This level of rigor isn’t optional when deploying an LLM feature at an enterprise customer with serious compliance requirements. You’re not just shipping code; you’re shipping a binary that will process their most sensitive data. Treat it like you’re shipping an OS kernel.

Monitoring: Proving the Negative (No Data Leaked)

Post-deployment, the real work begins. The security team didn’t just want a one-time review; they wanted continuous proof that the zero-egress property held. We instrumented three layers of defense:

• Network-Level: eBPF programs (via Cilium) that monitor all socket-level connections from the inference pod. Any unexpected outbound packet triggers an immediate alert and pod kill.
• Application-Level: The hash-logging sidecar mentioned earlier. We wrote a small Python service that consumed the vLLM request/response stream, computed hashes, and forwarded them to their Splunk instance. Zero raw text.
• Model-Level: Regular integrity checks on the model weights. A cron job in the pod recomputed the SHA256 of the model binary and compared it to the known-good value from deployment. Any drift (indicating tampering) would page us.

This is where the FDE role diverges from a pure software engineer. You’re writing the monitoring, writing the runbooks, and often sitting on the bridge call at 2 AM when an alert fires. You become the trusted technical counterpart to the customer’s security team. That trust is what converts a PoC into a $2M contract.

Career Context: Why FDEs Own the Security Conversation

If you’re an engineer looking to move into a Forward Deployed role, understand this: your ability to navigate enterprise security is worth more to an AI startup than your ability to fine-tune a model. Models are commoditizing. Trust isn’t.

At the time of this deployment, the median FDE comp (base + variable) for someone who could lead a security-hardened LLM deployment was $180K–$240K, with significant equity upside. The reason is simple: every AI startup selling to the enterprise hits this exact wall. The engineers who can get the feature past the CISO unlock revenue directly. You’re not a cost center; you’re the reason the deal closes.

If you’re building this skillset, start with the patterns in this case study. Practice containerizing an open-source model with zero egress. Write the NetworkPolicy. Build the hash-logging sidecar. When you can walk into a customer meeting and say, “Here’s how we run your data locally, here’s the cryptographic proof it never leaves, and here’s the packet capture to verify it,” you’ve moved from a builder to a deal-closer.

For more on the FDE career path and how AI-native startups use engineers to win enterprise deals, read our breakdown of how AI-native startups use Forward Deployed Engineers to win enterprise deals. And if you want to practice debugging in constrained environments (a core FDE skill), check out the playbook on debugging in the customer’s environment without direct access.

FAQ: Enterprise LLM Deployment

What’s the biggest mistake teams make when deploying an LLM feature at an enterprise customer? Assuming the security review will be a checkbox exercise. Most teams don’t realize that “no training on your data” clauses in API agreements don’t satisfy a serious InfoSec team. They want architectural proof, not legal promises.

Can I use a hosted model (like GPT-4) and still pass a strict review? It’s possible if your customer has an existing Azure or AWS relationship with the right contractual terms, but for financial services and defense, the trend is strongly toward self-hosted, open-weight models. If you must use an API, implement client-side encryption where you encrypt the prompt before it leaves your network and decrypt the response client-side.

How do you handle model updates in an air-gapped environment? The same way you deployed initially: signed containers, delivered via approved media. You never allow the inference pod to pull from the internet. This means your release cadence slows down, but the security team will accept that trade-off.

What open-source tools are essential for this pattern? vLLM or llama.cpp for serving, BGE or Instructor embeddings for RAG, Cosign for signing, Cilium for network policies, and a minimal Python/Go sidecar for hash logging. You don’t need a complex platform; you need composable, auditable pieces.

How does this relate to building agents in enterprise environments? The same zero-egress principles apply, but agents introduce more risk because they take actions. If you’re building an agent that negotiates meeting times or summarizes incidents, start with a read-only, local-only deployment pattern. For an example of a self-hosted agent workflow, see our guide on building an on-call incident summarizer that reads logs and drafts a postmortem with free LLMs.