Case Study: Deploying an LLM Feature at a Regulated Enterprise with Strict Air-Gap Rules

The Scenario: No Inbound, No Outbound, No Excuses

You are a Forward Deployed Engineer at a Series-C infrastructure company. Your customer is a defense-adjacent logistics contractor operating under strict regulatory frameworks. They handle Controlled Unclassified Information (CUI) and are subject to frameworks equivalent to ITAR and NIST 800-171. Their data center is effectively a SCIF: no persistent internet connectivity, no outbound API calls, and strict physical access controls.

The ask is deceptively simple: “We have 15,000 internal technical manuals. Our engineers spend 30% of their day searching PDFs. Give us an internal ChatGPT that runs completely offline.”

This is not a hypothetical. It is the exact class of problem that separates FDEs from standard Solutions Architects. You are not selling a cloud API key. You are shipping a living system into a black box.

Architectural Constraints and the 'Approved Components' List

Before writing a single line of code, you sit down with the customer’s security engineering team. They hand you a spreadsheet of “Approved Software Components.” Anything running on their metal must come from this list or pass a 6-week security audit.

The constraints are non-negotiable:

• No Docker Hub: Zero external image pulls allowed during deployment.
• No Telemetry: Every tool must be compiled or configured to block phone-home behavior.
• Immutable Infrastructure: The production server runs a hardened RHEL 8 STIG image.
• Air-Gapped Python: PyPI is a distant memory. Dependencies must be packed as wheel files on a clean machine and physically transferred.

Given these constraints, the modern “vibe-coded” cloud stack collapses. You cannot use OpenAI, Anthropic, or any managed embedding service. You need a local inference engine, a local vector store, and a dead-simple frontend that runs in a browser on localhost.

We settle on a stack that passes the initial technical sniff test:

• Inference: Ollama with a Mistral 7B quantized model (GGUF).
• Vector Store: ChromaDB in persistent mode.
• Embeddings: nomic-embed-text via Ollama.
• Ingestion: A Python script using PyMuPDF for extraction and LangChain’s text splitters.
• Backend: FastAPI served via Uvicorn, bound to localhost only.
• Frontend: A vanilla HTML/JS chat interface served from the same FastAPI process.

The Build: Local RAG with Ollama, Chroma, and a Python Shim

We build the ingestion pipeline first. The customer provides a sample of 500 technical manuals as PDFs. The ingestion script runs on a developer laptop (with internet) to resolve dependencies, then we freeze everything.

The core retrieval logic is standard RAG, but the engineering lies in the constraints. We cannot use LangChain’s default OpenAI embeddings. We swap to the Ollama embeddings endpoint:

from langchain_community.embeddings import OllamaEmbeddings
from langchain_community.vectorstores import Chroma

embeddings = OllamaEmbeddings(model="nomic-embed-text")
vectordb = Chroma(persist_directory="./chroma_db", embedding_function=embeddings)

We write a strict requirements.txt and use pip download to collect all wheels. We verify every transitive dependency against the approved components list. Three libraries are flagged for containing telemetry code. We fork them, comment out the offending requests.post calls, and rebuild the wheels. This is not glamorous work, but it is the work.

For the frontend, we avoid React or any Node.js build step. The customer’s security team has never approved an npm audit for an air-gapped system. We ship a single index.html with a <script> block that calls our FastAPI /chat endpoint. It is 200 lines of vanilla JavaScript. It works.

The Security Review: SBOMs, Data Flow Diagrams, and the 'No Telemetry' Rule

The security review is a 3-week gauntlet. We submit a Software Bill of Materials (SBOM) generated by syft for every component. We provide a 12-page data flow diagram showing that no packet can leave the localhost interface.

The biggest hurdle is Ollama. The security team discovers that Ollama, by default, attempts to pull models from its registry on startup. This is a hard fail. We modify the systemd service file to pass OLLAMA_MODELS=/opt/ollama/models and pre-load the Mistral GGUF directly from disk. We also set OLLAMA_HOST=127.0.0.1 and OLLAMA_ORIGINS="" to lock down the API.

We prove the system is truly air-gapped by running a tcpdump on the loopback interface during a 2-hour soak test. Zero packets leave the machine. The security team signs off.

Testing in a Simulated SCIF with No Internet

We cannot test on the customer’s actual hardware until the final deployment day. We simulate the environment on a spare Intel NUC in our office. We disable the WiFi card in BIOS, unplug the Ethernet cable, and install the exact RHEL 8 STIG image.

We transfer the wheels and models via a USB drive. We run the install script. It fails immediately. A missing system library (libstdc++ version mismatch) breaks the ChromaDB SQLite backend. We spend 4 hours debugging, then add a pre-install script that bundles the required .so files. We update the SBOM and re-submit for a delta review.

This is the FDE reality: the technical challenge is not the LLM. The challenge is the 47th dependency of a C++ library that nobody thought about until the air-gapped machine refused to cooperate.

Deployment, Validation, and the Awkward 'Sneakernet' Update

Deployment day arrives. We arrive at the customer’s facility at 0600. We pass physical security, surrender our phones, and enter the data center. The server is a 4U rack mount with an A100 GPU. We mount the USB drive, run the install script, and start the service.

The first query: “What is the torque specification for the XR-12 rotor assembly?” The system retrieves the correct manual section and generates a coherent answer in 3.2 seconds. The lead engineer stares at the screen. He types six more queries. Each one returns a correct, sourced answer. He turns to us and says, “This is going to save us 200 hours a month.”

Then comes the awkward part. He asks, “How do we update the manuals?” We explain the Sneakernet pipeline: new PDFs go on a USB drive, the ingestion script runs locally, and the ChromaDB is re-indexed. It is clunky, but it is secure. We leave behind a laminated runbook.

Career Context: Why Air-Gap Deployments Command Premium Comp

This case study is not just a technical story. It is a comp story. FDEs who can operate in regulated, air-gapped environments are rare. The skill set combines deep Linux administration, security compliance fluency, and the ability to ship product without the crutch of managed cloud services.

At the upper end of the market, FDEs with a track record of successful air-gapped deployments command a significant premium. For a detailed breakdown of the numbers, see our FDE Compensation Bands and How to Negotiate Your Offer in 2025. The delta between a standard cloud-only FDE and one who can deliver in a SCIF can be $50-80k in base salary alone, with higher equity upside because you are unlocking net-new revenue from customers who cannot use SaaS.

This is also where the Forward Deployed Engineer vs Consultant distinction becomes concrete. A consultant writes a whitepaper about air-gapped AI. An FDE hands the customer a USB drive with working code and sits in the SCIF until the query returns. Ownership is the difference.

For engineers looking to build the skills that lead to these deployments, start with projects that force you to run inference entirely locally. Our guide on GPU Passthrough on macOS VMs is a solid entry point for understanding the hardware layer. From there, graduate to building a full RAG system that never touches the internet, similar to the Codebase Q&A Bot with Gemini RAG but adapted for local-only execution.

FAQ: Air-Gapped LLM Deployments

Q: What is an air-gapped environment? A: A network or system physically isolated from unsecured networks, typically the public internet. Common in defense, critical infrastructure, and financial systems handling classified or sensitive data.

Q: Why not just use a VPN and a cloud LLM API? A: Regulatory frameworks like ITAR, NIST 800-171, and CMMC often prohibit sending sensitive data to external endpoints, even over encrypted tunnels. The data must stay on-premises at all times.

Q: What is the minimum hardware for a local LLM in an air gap? A: For a 7B parameter model with reasonable latency, a server with an NVIDIA A10 or A100 GPU and 32GB of system RAM is a practical minimum. CPU-only inference with quantization is possible but slow for interactive use.

Q: How do you keep the model updated? A: Model updates follow the same Sneakernet process as data. New GGUF files are transferred via physical media, validated against a checksum, and loaded manually. There is no auto-update.

Q: What is the biggest pitfall in air-gap deployments? A: Underestimating the dependency graph. A Python package that works on your internet-connected laptop may pull in 200 transitive dependencies, any one of which could phone home or require a system library absent from the hardened OS image. Always test on a true air-gapped clone before deployment day.

Q: How does this affect my career as an FDE? A: Air-gap competency is a high-leverage specialization. It opens doors to defense, intelligence, and critical infrastructure accounts that SaaS-only competitors cannot touch. It also builds a deep, transferable skill set in systems engineering that pays dividends throughout your career.