GPU Passthrough on macOS VMs: Fast llama.cpp Inference on Apple Silicon

The Breakthrough: Not Just Another Virtual GPU

For years, running a macOS virtual machine meant accepting a brutal trade-off. You got a pristine sandbox for testing, building, or isolating workloads, but you paid for it with anemic graphics performance. The VM saw a generic framebuffer. For LLM inference, this was a non-starter. You were stuck with CPU-only llama.cpp runs that turned a capable Apple Silicon chip into a sluggish mess.

That changed. The team at Cua has shipped a macOS VM that passes the host Apple Silicon GPU directly to the guest operating system. Not a paravirtualized driver. Not a software renderer. A genuine, Metal-accessible GPU that llama.cpp can see and use for hardware-accelerated inference.

The source material from Cua's engineering blog lays out the raw numbers: a Cua VM running llama.cpp with full GPU acceleration achieves token generation speeds within single-digit percentages of bare-metal performance. For an M-series chip, that means you can run a 7B parameter model inside a fully isolated VM at 30+ tokens per second instead of watching a CPU-only run crawl at 4-5 tokens per second.

This isn't a minor driver update. It's a fundamental shift in what a macOS VM is good for. The sandbox is no longer a penalty box for compute-heavy AI work.

Why This Matters for the Forward-Deployed Engineer

Forward Deployed Engineers live in the messy space between pristine demos and real customer environments. You're often asked to prototype an LLM feature locally, then ship it into an environment you don't fully control. The macOS VM with GPU passthrough solves three distinct FDE pain points.

Reproducible AI Development Environments. You can now snapshot an entire macOS VM—OS version, Python dependencies, Metal libraries, and the exact model weights—and hand that disk image to another engineer. They boot it, and llama.cpp sees the same GPU acceleration you had. No more "it works on my machine" when the machine in question is a specific MacBook Pro with a specific macOS point release.

Safe On-Device Inference Testing. Many enterprise deployments require local inference for data residency or air-gap reasons. Testing an on-device LLM feature means installing experimental Metal shader compilers or custom quantization kernels that can destabilize your main workstation. An FDE can now test these inside a throwaway VM that has full access to the GPU. If the kernel panics, you delete the VM, not your afternoon. For more on navigating enterprise constraints, see our case study on deploying an LLM feature with strict air-gap rules.

Benchmarking Without Bare-Metal Access. When a customer asks "how fast will this model run on our fleet of M2 Ultra Mac Studios?" and you don't have one sitting on your desk, a GPU-passthrough VM on your own M-series hardware gives you a credible lower bound. The virtualization overhead is small enough that your numbers translate. This is the kind of rapid, credible estimation that separates an effective FDE from a consultant who only delivers slideware. The distinction matters—we've written about the operating model differences between FDEs and consultants and ownership is the core of it.

The Architecture: How the GPU Passthrough Actually Works

To understand why this is hard, you need to look at Apple's I/O architecture. On Apple Silicon, the GPU isn't a discrete PCIe device you can hand to a hypervisor with a standard IOMMU mapping. It's a tightly coupled block on the M-series SoC, sharing a unified memory controller with the CPU and Neural Engine.

Cua's approach bypasses Apple's standard virtualization framework where it matters. Here's the simplified data flow:

The key insight is in the memory mapping. Instead of the hypervisor copying GPU command buffers between host and guest address spaces, Cua maps guest physical memory directly into a region the host GPU can see. The guest's Metal driver thinks it's talking to a real GPU. The command streams land directly in unified memory that the physical GPU cores consume.

For llama.cpp specifically, this means the Metal backend—which uses MTLBuffer objects for model weights and MTLComputeCommandEncoder for shader dispatch—operates without translation. The guest VM's Metal shader compiler generates the exact same GPU binaries it would on bare metal, and those binaries execute on the same physical cores.

The overhead that remains comes from two sources: the hypervisor's trap-and-emulate handling of certain privileged GPU register accesses during setup, and the fact that the guest VM's memory is a subset of physical memory, which can affect the GPU's cache replacement policy slightly. Neither is significant for the long-running, compute-bound shader dispatches that dominate LLM inference.

Benchmarks: Native vs. VM Inference Speed

Cua's published benchmarks tell a clear story. Running llama.cpp with a 7B Q4_K_M quantized model on an M2 Max:

Two things jump out. First, the GPU passthrough overhead is real but small—roughly 4% on token generation. For an interactive chat application, a human won't perceive the difference between 35.7 and 34.2 tokens per second. Both feel instantaneous.

Second, the CPU-only VM is an order of magnitude slower. This is the old world. If you're building a RAG pipeline that needs to run embedding generation or reranking inside a VM, CPU-only was your only option until now. That 89% performance cliff made VM-based AI workflows a non-starter.

The benchmark also reveals something about prompt evaluation. The 3.2% overhead on prompt processing is slightly lower than the generation overhead. This makes sense: prompt evaluation is a single large matrix multiply that saturates the GPU's compute units, leaving less idle time where virtualization overhead could interleave. Token generation, with its autoregressive loop of smaller operations, gives the hypervisor more chances to intervene between dispatches.

If you're building a codebase Q&A bot that ingests large context windows, the low prompt eval overhead matters. We've covered building a codebase Q&A bot with RAG using a different stack, but the architecture pattern transfers directly to a llama.cpp + Metal backend running inside a Cua VM.

Getting Hands-On: Spinning Up Your Own Accelerated VM

The quickest path from zero to inference:

1. Install Cua. The CLI is available via Homebrew: brew install trycua/cua/cua
2. Pull a macOS image. cua pull macos-sequoia-vanilla gets you a clean Sequoia VM.
3. Boot with GPU passthrough. cua run --gpu-passthrough macos-sequoia-vanilla starts the VM with the GPU exposed.
4. Inside the VM, clone and build llama.cpp with Metal support. The standard CMake build works: cmake -B build -DLLAMA_METAL=ON && cmake --build build --config Release
5. Download a GGUF model and run. ./build/bin/llama-cli -m models/7b-q4.gguf -p "Hello" -ngl 99

That -ngl 99 flag is critical. It tells llama.cpp to offload all layers to the GPU. Without it, you'll get a mixed CPU/GPU run that doesn't reflect the true passthrough capability.

The VM's disk image is a standard sparse bundle. You can preload it with models, Python environments, and scripts, then compress and distribute it to your team. This is the "golden image" pattern that enterprise deployment teams love—and that FDEs can use to guarantee reproducibility across customer PoCs.

For a practical example of automating LLM workflows that could run inside such a VM, look at our guide on building a daily standup bot with a free LLM. The same pattern of scheduled inference jobs translates directly to a VM-hosted deployment.

The Balanced Take: Where This Shines and Where It Doesn't

Let's be direct about trade-offs.

Where it shines:

• Reproducible AI dev environments. Snapshot a VM with exact dependency versions and share it. No more Metal version mismatches breaking your inference pipeline.
• Isolated testing of experimental inference stacks. Test new quantization types or custom Metal shaders without risking your main machine's stability.
• CI/CD for on-device AI. Run llama.cpp inference benchmarks in a GitHub Actions runner that's actually a Mac with Cua, getting GPU-accurate performance numbers in your CI pipeline.
• Customer PoCs on your own hardware. Demonstrate an air-gapped LLM feature running at full speed on a VM that mimics the customer's target Mac, without shipping them hardware.

Where it doesn't (yet) fit:

• Production serving. A VM adds management overhead. If you're deploying a production inference server on a dedicated Mac, run bare metal. The 4% overhead isn't worth the operational complexity.
• Multi-GPU or eGPU setups. Apple Silicon's unified memory architecture means there's only one "GPU" to pass through. If you're hoping to aggregate multiple Macs into a single virtual inference cluster, this isn't that.
• Training workloads. llama.cpp is inference-focused. If you need to fine-tune with MLX or PyTorch inside the VM, GPU passthrough should work in principle, but the memory overhead of virtualization might push large models out of the unified memory budget.

The technology is genuinely impressive, but it's a tool for a specific job: giving engineers a fast, isolated, reproducible environment for on-device AI work. It's not a replacement for bare-metal inference servers, and it's not a way to magically pool GPU resources across machines.

FAQ

Does this work with any Apple Silicon Mac? Yes, any M1 or later chip. The GPU passthrough leverages the same hardware capabilities across the M-series lineup. Performance scales with GPU core count just as it does on bare metal.

Can I run multiple GPU-accelerated VMs simultaneously? You can run multiple VMs, but the GPU is not partitioned. One VM gets GPU access at a time through the passthrough mechanism. For concurrent workloads, you'd need to run inference inside a single VM or use multiple physical machines.

Does this require a specific macOS version in the guest? The guest should run macOS 14 (Sonoma) or later to have the Metal driver support that Cua's passthrough expects. Older guest OS versions may not recognize the virtualized GPU device.

Is this the same as PCIe GPU passthrough on Linux/KVM? Conceptually similar, but the implementation is entirely different. Apple Silicon's unified memory architecture means there's no PCIe topology to virtualize. Cua's approach is specific to the M-series SoC design.

Can I use this for non-AI GPU workloads? Yes. Any Metal-compatible application running in the guest will see GPU acceleration. This includes video encoding, 3D rendering, and other compute tasks. The passthrough is not specific to llama.cpp.

What's the memory overhead of the VM itself? The VM's macOS guest consumes roughly 4-6 GB of RAM for the operating system. The remaining unified memory is available for model weights. On a 32 GB Mac, expect about 26-28 GB usable for inference inside the VM.

Will this work with MLX or only llama.cpp? The GPU passthrough exposes a standard Metal device, so any framework that uses Metal—including MLX, PyTorch with MPS backend, and Core ML—should work. The Cua team's benchmarks focus on llama.cpp because it's the most common inference engine, but the capability is general-purpose.