How AirLLM Runs 70B Models on a 4GB GPU with Layer-Wise Loading

What Happened: Running Giants on a Potato

The open-source project AirLLM demonstrated something that breaks a lot of engineers' mental models: running a 70-billion-parameter large language model (think Llama 2 70B) on a single consumer GPU with just 4GB of VRAM. The same technique works for even larger models like Llama 3 405B on a single 16GB card.

This isn't quantization magic or a distilled student model. It's the full-fat, original-precision model. The trick is a strategy called layer-wise loading, which is fundamentally different from how most people think about model inference.

Normally, when you load a model, the entire computation graph sits in GPU memory. A 70B parameter model in 16-bit precision needs roughly 140GB of VRAM. That's four A100-40GB cards. But AirLLM loads exactly one transformer layer at a time, runs the forward pass for that layer, offloads the activations to CPU RAM, and then loads the next layer. Rinse and repeat for all 80 layers of a typical 70B model.

The result is a system that can generate a few tokens per minute on hardware that costs less than a nice dinner. It's not fast, but it works. For engineers who've been locked out of large-model experimentation because they don't have a cluster, this is a door swinging open.

The Engineering Mechanics: It's All About the Layers

To understand why this works, you need to look at the anatomy of a transformer model. A 70B parameter LLM isn't one monolithic blob. It's a stack of identical transformer blocks—typically 80 layers for Llama 2 70B, each containing attention mechanisms, feed-forward networks, and layer normalization.

The key insight: inference is sequential through these layers. You don't need layer 47 until you've finished computing layer 46. The forward pass is a pipeline. AirLLM exploits this by treating each layer as an independent unit that can be loaded, executed, and discarded.

Here's what the data flow looks like under the hood:

This isn't entirely novel. CPU offloading has existed in various forms (DeepSpeed ZeRO-Offload, llama.cpp's mmap mode). But AirLLM's implementation is particularly aggressive about minimizing the GPU footprint at any given moment. It only keeps the current layer's weights plus the intermediate activations in VRAM. Everything else lives in system RAM or on disk.

The tradeoff is blindingly obvious: you're moving gigabytes of data across the PCIe bus for every single layer transition. A 70B model with 80 layers means roughly 80 round-trips of weight transfer per token generated. That's the bottleneck.

Memory Math: Why 4GB Suddenly Works

Let's do the numbers that make this possible. A Llama 2 70B model has approximately 70 billion parameters. In float16 (2 bytes per parameter), that's 140GB of weights. Divide by 80 layers, and each layer is roughly 1.75GB of weights.

But you also need memory for:

• KV cache: The key-value pairs from attention that grow linearly with sequence length
• Activations: The intermediate tensors flowing through the network
• Overhead: CUDA context, framework buffers, etc.

AirLLM keeps the KV cache in CPU memory and only moves the current layer's weights plus the immediate activations onto the GPU. The per-layer GPU memory requirement looks roughly like:

That fits comfortably in 4GB of VRAM, with breathing room. The KV cache for a 2048-token context in float16 adds another ~1.3GB, but AirLLM parks that in CPU RAM, not GPU memory.

This is the core engineering tradeoff: you're swapping compute for memory bandwidth. The GPU is constantly waiting on data from system RAM. For a single token generation on a 70B model, you're transferring roughly 140GB of weights (80 layers × 1.75GB) across the PCIe bus. Even at PCIe 4.0 x16 speeds (~32 GB/s theoretical), that's at least 4-5 seconds of pure data transfer per token, before any actual computation.

Why This Matters for FDEs and Working Engineers

For a Forward Deployed Engineer, the ability to run large models locally changes the prototyping calculus. Here's the real-world impact:

Offline-first demos become possible. When you're deploying to a customer site with air-gapped environments or strict data residency requirements, shipping a 4GB GPU machine that can run a 70B model—even slowly—is a powerful capability. You can build proof-of-concept integrations without cloud dependencies. This aligns directly with the FDE toolkit philosophy of shipping fast with minimal infrastructure.

Experimentation without budget approval. Getting access to A100 clusters often requires procurement cycles, budget sign-offs, and cloud quota requests. A 4GB GPU is a gaming laptop. You can experiment with prompt engineering, fine-tuning strategies, and model behavior on the actual 70B model—not a quantized 7B approximation—before committing to expensive compute. This is the kind of high-leverage skill that separates senior FDEs from the pack.

Understanding model behavior at scale. There's a qualitative difference between how a 7B model and a 70B model handle complex instructions. If you're building an agent that needs to follow multi-step reasoning, testing against the real model matters. AirLLM lets you validate behavior on actual large models before optimizing for production inference.

The portfolio angle. Demonstrating that you can deploy and run a 70B model on constrained hardware is exactly the kind of project that stands out in an FDE portfolio. It shows you understand memory hierarchies, inference optimization, and the practical constraints of real-world deployment.

How to Try It Today: A Minimal Setup

Getting AirLLM running is straightforward. Here's the minimal path for a Llama 2 70B model on a machine with 4GB+ VRAM and at least 32GB of system RAM (you'll want 64GB+ for comfort):

# Install AirLLM
pip install airllm

# If you have the model weights locally (HuggingFace format)
# Assuming model is at ./Llama-2-70b-hf

from airllm import AirLLMLlama2

# Initialize with layer-wise loading
model = AirLLMLlama2(
    "meta-llama/Llama-2-70b-hf",
    compression="none",  # full precision, no quantization
)

# Generate with the model
input_text = "Explain the concept of layer-wise model loading:"
output = model.generate(input_text, max_new_tokens=50)
print(output)

A few things to expect on your first run:

• First token latency is brutal. The initial prefill pass has to process your entire input prompt through all 80 layers. Expect 30-120 seconds before you see the first token.
• Subsequent tokens are faster but still slow. Each additional token requires a full pass through all layers, meaning another 80 weight transfers. You're looking at 5-15 seconds per token on consumer hardware.
• System RAM is the hidden constraint. Your CPU RAM needs to hold the entire model (140GB for 70B float16) plus the KV cache. If you're running 32GB of RAM, you'll need to use 4-bit quantization (compression="4bit") to fit.

For practical experimentation, consider using the quantized mode:

model = AirLLMLlama2(
    "meta-llama/Llama-2-70b-hf",
    compression="4bit",  # 4-bit quantization, ~35GB total
)

This brings the model size down to ~35GB, fitting in 64GB of system RAM with room for the KV cache. The quality degradation is noticeable but often acceptable for prototyping.

The Balanced Take: When It's Magic, When It's Painful

Let's be honest about the tradeoffs. AirLLM is not a production inference solution. It's a prototyping and experimentation tool that makes large models accessible on small hardware. Here's where it shines and where it falls apart:

Where it's genuinely useful:

• Validating prompts and chain-of-thought reasoning on full-scale models before deploying to expensive inference endpoints
• Building offline demos where response latency of 30-60 seconds is acceptable
• Educational exploration of large model behavior without cloud costs
• Testing model-specific quirks (tokenizer behavior, special token handling, output formatting) on the real model

Where it's painful:

• Any interactive use case. Nobody wants to wait 10 seconds per token in a chat interface.
• Batch processing or high-throughput scenarios. The sequential layer loading is inherently serial.
• Long context windows. The KV cache in CPU memory becomes a second bottleneck as context grows.
• Fine-tuning. While technically possible, the layer-wise approach makes gradient computation brutally slow.

The deeper lesson is about understanding the memory-compute tradeoff in ML systems. AirLLM is an extreme point on the spectrum: maximum memory efficiency, minimum compute efficiency. Most production systems sit somewhere in the middle, using techniques like tensor parallelism, pipeline parallelism, or KV cache quantization to balance the two.

For an FDE, the value isn't in using AirLLM as your daily driver. It's in understanding that these tradeoffs exist and knowing when to reach for which tool. The same layer-wise thinking applies to building agents that process documents or automating resume tailoring—you're always making decisions about what to keep in memory, what to offload, and what to recompute.

FAQ

Can AirLLM run on a machine with no GPU at all? Yes, but it will be extremely slow. AirLLM can fall back to CPU-only inference, but layer-wise loading on CPU means you're moving data within system RAM rather than across the PCIe bus. Expect 30-60 seconds per token for a 70B model.

How does this compare to llama.cpp's offloading? llama.cpp can offload a configurable number of layers to GPU while keeping the rest on CPU. AirLLM is more extreme—it only keeps one layer on GPU at a time. llama.cpp's approach is generally faster if you have more VRAM to play with, since it can keep multiple layers resident. AirLLM wins when VRAM is extremely constrained.

Does layer-wise loading affect output quality? No. The computation is mathematically identical to running the full model on a cluster. You're getting the exact same logits and token probabilities. The only difference is speed.

What about model formats? Does it only work with HuggingFace models? AirLLM primarily targets HuggingFace transformer models (Llama, Mistral, Qwen, etc.). Support varies by architecture. Check the project's model compatibility table before downloading 140GB of weights.

Is this useful for the Llama 3 405B model? Yes, with caveats. A 405B model in float16 is ~810GB. Even with 4-bit quantization (~200GB), you need serious system RAM (256GB+). But the layer-wise approach still works—each of the 126 layers in Llama 3 405B is roughly 6.4GB in float16, or 1.6GB in 4-bit. The limiting factor becomes system RAM capacity, not GPU VRAM.

Can I use this for fine-tuning or just inference? AirLLM is designed for inference. Fine-tuning with layer-wise loading is theoretically possible but impractical—backpropagation requires storing activations from all layers, which defeats the memory-saving purpose. For fine-tuning on constrained hardware, look into QLoRA or other parameter-efficient methods instead.