Running a 26B LLM on a 13-Year-Old CPU: The Inference Optimization Stack

The Absurd Benchmark

A 26-billion parameter large language model—Google’s Gemma 4—running inference at 5 tokens per second. No GPU. No NPU. No exotic ASIC. The hardware? A 13-year-old Intel Xeon E5-2690, a chip released in 2012 when the state-of-the-art in NLP was a bag-of-words classifier. The experiment, detailed by Neomind Labs, isn’t just a parlor trick. It’s a masterclass in inference optimization that exposes the real bottleneck in LLM deployment: memory bandwidth, not FLOPs.

5 tokens per second is borderline usable for a chat interface. It’s not snappy, but it’s not a slide deck either. For batch processing, document summarization, or overnight data extraction jobs, it’s genuinely practical. And it was achieved on a machine you could pull from an e-waste bin.

The key specs: dual Xeon E5-2690s (8 cores each, 16 threads total), 256GB of DDR3 ECC RAM, and zero accelerator cards. The model was Gemma 4 26B, a dense transformer. No sparsity tricks, no distillation. This is the full-fat model, just running in a severely memory-constrained environment.

The Optimization Stack (The Real Meat)

This isn’t one magic trick. It’s a layered stack of optimizations, each compounding on the last. Break any layer and you’re back to 0.1 tokens/sec, which is effectively dead. Here’s the architecture:

Quantization: The Non-Negotiable First Step

A 26B parameter model in FP16 consumes roughly 52GB of memory. That’s just the weights. Add the KV cache, activations, and overhead, and you’re looking at 60GB+ easily. The Xeon system has 256GB, so you might think it fits. But DDR3-1600 has a theoretical peak bandwidth of about 51 GB/s per channel, and real-world throughput is far lower.

The bottleneck in CPU inference is reading every single weight from RAM for every token generated. At FP16, you’re moving 52GB of weights through the memory bus for each token. On a system with ~30-40 GB/s of practical bandwidth, that’s 1.3-1.7 seconds per token—slower than a human typing.

Enter Q4_K_M quantization via llama.cpp. This squeezes the model to roughly 15GB. It uses a block-wise quantization scheme: groups of weights share a scaling factor, with a 4-bit integer representation. The K-quant variants (Q4_K_M specifically) apply higher precision to attention and feed-forward layers that are more sensitive to quantization error. The result? A model that’s 3.5x smaller with negligible perplexity degradation for most tasks.

Now you’re moving 15GB per token instead of 52GB. At the same memory bandwidth, that’s 0.4-0.5 seconds per token. We’re at 2 tokens/sec theoretically. The rest of the stack claws back the remaining 3x.

llama.cpp: The Inference Engine That Won

llama.cpp is the de facto standard for CPU inference, and for good reason. It’s a C++ implementation with no Python overhead, no framework tax, and aggressive kernel optimization. Key features it brings to this stack:

• mmap loading: The model file is memory-mapped directly into the process address space. No loading phase, no buffer copies. The OS pages in weights on demand. This also means multiple processes can share the same physical pages—critical if you’re running multiple inference workers.
• Thread-parallel evaluation: During prompt processing (the prefill phase), the model can evaluate all input tokens in parallel across CPU cores. This saturates the memory bus and gets the prompt through quickly.
• BLAS acceleration: Optional integration with OpenBLAS or Intel MKL for matrix multiplications, though on this Xeon the memory bandwidth is the ceiling, not compute.

NUMA Awareness: The Silent Killer

The dual-socket Xeon E5-2690 is a NUMA (Non-Uniform Memory Access) system. Each socket has its own memory controller and local RAM bank. A thread running on Socket 0 accessing memory attached to Socket 1 pays a latency penalty crossing the QPI (QuickPath Interconnect) link—roughly 2x the latency of local access.

If you naively run inference without pinning threads to cores and allocating memory from the correct NUMA node, you’re bleeding performance. The Neomind Labs setup used numactl to bind the llama.cpp process to specific NUMA nodes, ensuring threads only access local memory. On a single-socket consumer chip, this is irrelevant. On a dual Xeon, it’s worth 20-30% throughput.

KV Cache Management

The KV cache stores the key and value tensors for every previous token in the sequence. For a 4096-token context window on a 26B model, this can easily consume 2-4GB of memory. On a GPU, this lives in VRAM. On a CPU, it lives in DDR3.

llama.cpp allocates the KV cache in a contiguous block and reuses it across requests. The size is configurable. Too small and you truncate context. Too large and you waste precious memory bandwidth on cache lines that aren’t hot. The sweet spot depends on your use case—for a chatbot, 2048 tokens is plenty. For document QA, you might need 8192+.

The Token Generation Loop

This is where the architecture forces a serial bottleneck. Prompt processing (prefill) is parallel: all input tokens are processed simultaneously, saturating all cores and the memory bus. But token generation is autoregressive: each new token depends on all previous tokens. You can’t parallelize across the sequence dimension.

So the generation loop becomes a tight dance:

1. Read 15GB of quantized weights from RAM.
2. Compute the next token logits.
3. Sample a token.
4. Update the KV cache.
5. Repeat.

At 5 tokens/sec, each iteration takes 200ms. That’s 200ms of the CPU hammering the memory controllers while the cores wait. This is why CPU inference is memory-bound, not compute-bound. Adding more cores doesn’t help after a point—you just have more cores waiting on the same bus.

Why This Matters for Forward Deployed Engineers

Forward Deployed Engineers live in the gap between what a model can do in a research paper and what it can do in a customer’s environment. That environment is often air-gapped, compliance-locked, or running on hardware that was purchased before the transformer architecture was invented. This experiment is a blueprint for that gap.

The Real FDE Constraints

When you’re deploying an LLM at a hospital, a defense contractor, or a manufacturing plant, you face constraints that SaaS engineers never see:

• No cloud, ever. The data can’t leave the building. No AWS, no Azure, no API calls to OpenAI.
• No new hardware. The procurement cycle for a GPU server can be 18 months. The compliance review for a new device on the network can be 6 months. You ship on what’s already racked.
• No internet. The model weights have to be physically carried in on a drive.
• No Python 3.12. The server runs RHEL 7 with Python 3.6 and you can’t update it because the vendor’s FDA-validated software depends on that exact version.

This is the reality where a quantized Gemma 4 on llama.cpp isn’t a hobby project—it’s the only viable path to production. And 5 tokens/sec on a decade-old Xeon means you can deploy a capable LLM without touching the procurement org chart.

We’ve written extensively about what an FDE actually ships in a 60-hour week at an AI startup. This kind of optimization work—squeezing a model onto hostile hardware—is exactly the type of problem that fills those hours.

The Cost Math

Compare the options for on-prem, air-gapped deployment:

For batch workloads—nightly summarization of 10,000 support tickets, entity extraction from scanned documents, generating embeddings for a legacy document corpus—5 tokens/sec on existing hardware is often the right answer. Not because it’s fast, but because it exists.

The FDE Playbook: Steal This Stack

If you want to replicate or adapt this for a deployment, here’s the concrete recipe. This isn’t a tutorial, it’s the architectural decisions that matter.

Model Selection

Don’t default to the largest model you can fit. For CPU inference, smaller models with better quantization tolerance are your friend. Gemma 4 26B at Q4_K_M works because Google trained it with knowledge distillation, making it robust to precision loss. Not all models quantize equally well. Test perplexity on your specific task before committing.

For many FDE use cases, you don’t need 26B parameters. A Q8_0 quantized 7B model running at 15 tokens/sec on the same hardware will feel snappy and handle structured extraction, classification, and RAG retrieval just fine. We’ve covered building a fully local RAG chatbot over PDFs and notes with Ollama—that architecture pairs perfectly with this inference stack.

Quantization Level Selection

llama.cpp offers a bewildering array of quantization formats. Here’s the cheat sheet for CPU deployment:

The “_M” suffix means medium-sized LUTs (look-up tables). “_S” is smaller and faster but slightly lower quality. For a 26B model on CPU, Q4_K_M is the sweet spot. For a 7B model, you can afford Q6_K or Q8_0.

Context Window Sizing

The KV cache is a fixed allocation. If you set it to 32768 tokens but only use 2048, you’re wasting memory bandwidth on dead cache lines. Profile your actual usage. Most RAG and extraction tasks need 4096 or less. If you need long context, consider a sliding window attention pattern or a separate summarization step to compress history.

For an example of chaining models together to handle complex workflows, look at our multi-agent research assistant build with Gemini Flash. The same principle applies: don’t make one model do everything.

System Tuning Checklist

# 1. NUMA binding (dual-socket systems only)
numactl --cpunodebind=0 --membind=0 ./llama-cli ...

# 2. Thread count: match physical cores, not hyperthreads
# For 16 physical cores across 2 sockets:
./llama-cli -t 8 -tb 1 ...
# -t: total threads, -tb: threads per batch

# 3. Disable transparent hugepages (reduces latency jitter)
echo never > /sys/kernel/mm/transparent_hugepage/enabled

# 4. CPU governor to performance
echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor

A Balanced Take: When Not to Do This

This is an engineering marvel, but it’s not a universal solution. Here’s when you should look elsewhere:

Latency-sensitive interactive use. 5 tokens/sec means a 100-token response takes 20 seconds. For a customer-facing chatbot, that’s unacceptable. Users will assume it’s broken and refresh the page. For internal tools where the user has context and patience, it can work.

High-throughput serving. If you need to handle 100 concurrent requests, a single Xeon won’t cut it. You’d need to batch requests cleverly (continuous batching), which llama.cpp server mode supports but is still limited by memory bandwidth. A single GPU can serve orders of magnitude more throughput.

Models that quantize poorly. Some architectures—particularly MoE (Mixture of Experts) models—don’t quantize as gracefully as dense models. The expert routing can amplify quantization errors. Test before you commit.

When you can actually get a GPU. If the procurement cycle is 3 months and the project timeline is 6 months, it might be worth waiting. A used RTX 3090 with 24GB can run a Q4 quantized 26B model at 30+ tokens/sec and costs under $1,000. The engineering time you spend optimizing CPU inference might exceed the cost of the GPU.

But that last point misses the real FDE value. The skill of making something work on hostile hardware—understanding the memory hierarchy, profiling bottlenecks, stacking optimizations—is transferable. It makes you a better engineer when you do have a GPU. You’ll know what’s actually happening under the CUDA abstractions.

FAQ

Q: Can I do this on my laptop? A: Yes, with caveats. A modern laptop with 32GB of RAM can run a Q4_K_M quantized 7B model comfortably. A 26B model at Q4_K_M needs ~15GB for weights plus KV cache, so 32GB is tight but workable. Expect 2-4 tokens/sec on a recent Intel or AMD mobile chip. Apple Silicon with unified memory does significantly better—an M2 Max with 64GB can hit 15+ tokens/sec on the same model.

Q: Why not use a smaller model? A: For many tasks, you should. A Q8_0 7B model running at 10-15 tokens/sec on CPU will outperform a Q4_K_M 26B model at 5 tokens/sec for structured extraction and classification. The larger model only wins on tasks requiring deep reasoning, complex instruction following, or broad world knowledge. Profile your task.

Q: Does this work with any model? A: Any model supported by llama.cpp’s GGUF format. This covers Llama, Mistral, Gemma, Qwen, Phi, and most derivatives. Check the llama.cpp GitHub for the current support matrix. Vision models and audio models have additional requirements.

Q: Is 5 tokens/sec actually usable? A: For batch processing: absolutely. For interactive chat: borderline. The human reading speed for comprehension is about 5-10 tokens/sec, so the model is keeping pace with the reader. The issue is the initial latency before the first token appears. Use streaming output so the user sees tokens as they’re generated rather than waiting for the full response.

Q: What’s the biggest model that can run on CPU? A: The practical limit is your RAM capacity divided by the compression ratio of your chosen quantization. With Q2_K quantization, you can fit roughly 1 billion parameters per GB. A 256GB server could theoretically run a 200B+ model at Q2_K, but the quality degradation would be severe and the token rate would be glacial. The Neomind setup at 26B Q4_K_M is near the sweet spot for this class of hardware.