Dissecting vLLM: How PagedAttention and Continuous Batching Maximize Throughput

The Problem: GPU Memory as the Bottleneck

You spin up an open-source model like Llama 3 70B. You slap a FastAPI wrapper on it with a static batch size of 8. It works—until a user sends a 4,000-token prompt. The GPU runs out of memory, the request OOMs, and the whole batch collapses. You dial the batch size down to 2. Latency is fine, but your throughput is abysmal. You’re burning expensive A100 hours serving a trickle of tokens.

This is the exact pain point vLLM was built to solve. The core insight from the UC Berkeley team is that LLM serving is a memory management problem, not a compute problem. GPUs are fast at matrix math. What kills them is the Key-Value (KV) cache—the growing tensor that stores attention states for every token in every sequence.

In a naive serving system, you pre-allocate a contiguous block of memory for each request’s KV cache. You have to guess the maximum possible sequence length. If a user sends a short prompt, you waste 80% of that allocation. If they generate a long output, you hit the pre-allocated wall and crash. This internal fragmentation and reservation waste means a 70B model might only serve 4-6 requests concurrently on a single A100, even though the compute headroom could handle 20+.

This matters for engineers and FDEs because throughput directly maps to cost. If you’re building a product on top of LLMs—a coding assistant, a document summarizer, a multi-agent research tool like the one we walk through building with Gemini—every millisecond of idle GPU time is money leaking out of your burn rate. Understanding vLLM’s internals lets you reason about latency/throughput tradeoffs and debug OOMs that aren’t really OOMs.

PagedAttention: Virtual Memory for KV Cache

vLLM’s first breakthrough is treating the KV cache the way operating systems treat physical memory. Instead of a single monolithic block per request, the KV cache is split into fixed-size blocks (think 16 or 32 tokens per block). These blocks don’t need to be contiguous. A logical sequence maps to a linked list of physical blocks, exactly like virtual memory pages map to physical frames.

Why does this matter? Three words: zero memory waste. When a request finishes, its blocks are immediately freed and handed to the next waiting request. When a request generates a new token and needs more space, it grabs any free block from the pool. No pre-allocation guesswork. No internal fragmentation from padding short sequences to the max length.

But the real impact is on sharing. In techniques like beam search or parallel sampling, multiple output sequences share the same prompt tokens. In a naive system, you copy the entire KV cache for each sequence. With PagedAttention, the block table simply points multiple logical sequences to the same physical blocks for the shared prefix. Copy-on-write semantics apply: only when a sequence diverges does it allocate new blocks. This is a massive memory savings for any workload that reuses prompts—which is practically every production LLM application.

The engineering detail that makes this work is the custom CUDA kernel. Standard attention implementations expect a contiguous KV cache. PagedAttention’s kernel walks the block table at runtime, gathering KV states from scattered physical blocks into the attention computation. This adds a small overhead, but the memory efficiency gain more than compensates.

Continuous Batching: Breaking the Sequence Barrier

Static batching waits for all requests in a batch to finish before admitting new ones. If one request generates a 500-token essay and another finishes in 10 tokens, the GPU idles waiting for the slowpoke. This is the "bubble" problem.

Continuous batching—vLLM’s second major innovation—operates at the iteration level, not the request level. An iteration is a single forward pass that generates one token per active sequence. After each iteration, the scheduler can:

1. Eject finished sequences immediately.
2. Admit new waiting requests into the freed slots.
3. Preempt long-running sequences if memory pressure spikes.

This means the GPU is never waiting. The batch is always full of active work. For a typical chat application with mixed prompt lengths, continuous batching can deliver 2-10x higher throughput than static batching at the same latency SLA.

Preemption is where it gets interesting. If memory fills up and a high-priority request arrives, vLLM can swap a running sequence’s KV cache blocks to CPU RAM, free the GPU blocks, and restore them later. This is virtual memory swapping, applied to attention states. The swapped sequence pauses, the urgent request runs, and the paused sequence resumes without losing its generation context.

For FDEs building demos or prototypes, this matters because it means you can run larger models on smaller GPUs. You can serve a 70B model on a single A100 with reasonable concurrency by leveraging swap space. The latency hit on swapped sequences is noticeable but often acceptable for batch processing or internal tools.

How to Actually Use vLLM Today

vLLM is not a research paper—it’s a pip-installable Python package with a production-ready OpenAI-compatible API server.

pip install vllm

Spin up a server in one command:

python -m vllm.entrypoints.openai.api_server \
  --model meta-llama/Meta-Llama-3-8B-Instruct \
  --max-model-len 8192 \
  --gpu-memory-utilization 0.95 \
  --max-num-seqs 64

--max-num-seqs controls the maximum concurrent sequences in the continuous batch. Set this high (64-128) for throughput-optimized workloads. Dial it down (8-16) for latency-sensitive interactive chat.

--gpu-memory-utilization tells vLLM how much VRAM it can use. The default 0.90 leaves headroom for CUDA context overhead. Push it to 0.95 if you’re running headless on a dedicated instance.

The server exposes an /v1/chat/completions endpoint that drop-in replaces your OpenAI client code. If you’ve built a Twitter thread writer with Groq or a flashcard generator with Ollama, swapping the base URL to http://localhost:8000/v1 is all it takes.

For programmatic use, the LLM class gives you direct control:

from vllm import LLM, SamplingParams

llm = LLM(model="meta-llama/Meta-Llama-3-8B-Instruct")
sampling_params = SamplingParams(temperature=0.8, top_p=0.95, max_tokens=256)
prompts = ["Summarize the PagedAttention paper.", "Write a haiku about GPUs."]
outputs = llm.generate(prompts, sampling_params)

The engine handles batching internally. You just feed it a list of prompts and it saturates the GPU.

Quantization is a first-class citizen. Pass --quantization awq or --quantization gptq to run 4-bit quantized models. A 70B model at 4-bit fits comfortably on a single A100 with room for a large KV cache pool.

A Balanced Take: The Sharp Edges

vLLM is not a silver bullet. Here’s what bites engineers in production.

Startup time is brutal. Loading a 70B model with vLLM takes 2-5 minutes depending on disk speed. It pre-allocates the KV cache pool, compiles CUDA kernels, and warms up the attention backend. This is fine for long-running servers but painful for CI/CD pipelines or spot-instance preemption scenarios. You’ll want health checks with generous timeouts.

Memory fragmentation still exists. While PagedAttention eliminates internal fragmentation, external fragmentation can occur. If the block allocator can’t find enough contiguous logical slots in the scheduler, you’ll see No available memory errors even with free blocks scattered around. The --max-num-seqs and --max-num-batched-tokens flags interact in non-obvious ways. Tuning these for your specific workload distribution is an art.

Custom models need adapter work. vLLM supports most HuggingFace architectures out of the box, but if you’ve fine-tuned a model with custom attention patterns or exotic position embeddings, expect to write a model adapter. The PagedAttention kernel assumes a specific KV cache layout. Deviate from it, and you’re in for a debugging session.

Speculative decoding is still maturing. While vLLM supports it, the integration with continuous batching is complex. Draft model execution and target model verification need careful scheduling to avoid pipeline bubbles. If you’re chasing sub-20ms time-to-first-token, you may need to go deeper than the default config.

For FDEs specifically: vLLM is a fantastic tool for building demos that feel production-grade. A single A100 running vLLM can serve a dozen concurrent users with sub-second latency on a 13B model. This lets you build interactive prototypes—like a personal finance categorizer that processes bank statements in real-time—without waiting for inference. The OpenAI-compatible API means your demo code doesn’t change when you move to production infrastructure.

The balanced view: vLLM is the default choice for self-hosted LLM serving in 2025. It’s not perfect, but it’s the closest thing to a reference implementation of efficient LLM inference. The paper’s ideas—PagedAttention and continuous batching—have been absorbed by virtually every inference framework since. Understanding them is now table stakes for anyone deploying LLMs.

FAQ

Q: Does vLLM work with non-NVIDIA GPUs?

A: As of 2025, vLLM supports AMD ROCm and Intel Gaudi accelerators, though the CUDA backend is the most mature. Apple Silicon via Metal is experimental. Expect to spend time on driver compatibility for non-CUDA paths.

Q: How does vLLM compare to TensorRT-LLM?

A: TensorRT-LLM uses ahead-of-time compilation and can squeeze out lower latency for fixed model architectures. vLLM trades a small latency overhead for zero-compilation deployment and dynamic scheduling flexibility. For rapid iteration, vLLM wins. For peak throughput on a frozen model, TensorRT-LLM often edges ahead.

Q: Can I use vLLM with LoRA adapters?

A: Yes. vLLM supports multi-LoRA serving, where a single base model can serve requests with different LoRA adapters in the same batch. The adapter weights are loaded into GPU memory and switched per-request with minimal overhead.

Q: What’s the minimum GPU for running a 7B model?

A: A single RTX 3090 or 4090 with 24GB VRAM comfortably runs 7B models at FP16 with vLLM. With 4-bit quantization, you can run 13B models on the same hardware. The PagedAttention memory savings really shine on these consumer GPUs where every megabyte counts.

Q: How do I debug OOM errors in vLLM?

A: First, check --gpu-memory-utilization. If it’s at 0.90, try 0.85. Second, reduce --max-num-seqs—fewer concurrent sequences means smaller KV cache pool. Third, lower --max-model-len if your use case doesn’t need full context. Finally, enable swap space with --swap-space 4 (in GB) to spill KV cache blocks to CPU RAM under pressure.

For a deeper dive into the original architecture, see Aleksa Gordić’s detailed breakdown.