Track 09 · Systems & unit costLesson 07 of 11
Making it feel fast
Serving and latency work, because perceived speed drives adoption.
- Lesson library · reference
- 35 min read
- 60 min hands-on
- Pro
What you'll be able to do
01- You can explain why prefill and decode have different bottlenecks and map them to time to first token (TTFT) and inter-token latency (ITL).
- You can calculate whether a model fits on a GPU configuration from weights plus KV cache, and say what to change when it does not.
- You can choose serving features (batching, paged KV, prefix caching, speculative decoding, quantisation, parallelism) for a latency target.
- You can pull the cheapest lever first: application, then provider, then serving stack.
- You can design and defend a latency measurement.
Before you start
- Tokens, context windows and why answers go wrong
- Self-hosted, VPC and air-gapped: how self-hosted weights get deployed; this file covers how they get served fast.
- SLOs and latency budgets
- Cost per resolved case
The short version
Prefill reads the whole prompt in parallel and produces the first token; decode produces every later token one step at a time. Users feel TTFT and ITL; agents feel end-to-end time. Engines get speed from batching, paged KV memory, prefix reuse, speculative drafts, fewer bits and multi-GPU splits. The application usually has bigger wins first: stream, emit fewer tokens, make fewer round trips, parallelise, and use small models for small steps.
- E2E ≈ TTFT + (output tokens − 1) × ITL. Output length is often the biggest term you control.
- Decode is memory-bandwidth bound, so batching raises throughput a lot and per-user latency a little, until memory runs out.
- GPU memory = weights + KV cache + reserve. The KV cache decides how many users fit.
Pro lesson
The rest of this lesson is in Pro.
The mental model, worked example, field note, failure table, exercise and self-check, with the voice tutor.
- Every stage and specialisation, in full
- Every lab and every role project
- Halden weeks 2–10, on your record
- Every drill, live in the playground
- All solutions, plus the server judge
- Lab Assistant and every Teach with AI