Track 01 · Models & contextLesson 01 of 10
Tokens, context windows and why answers go wrong
Read a usage block and a stop reason, and explain the failure before the customer asks.
- The FDE path · Stage 2 of 6
- 35 min read
- 45 min hands-on
- Pro
What you'll be able to do
01- Explain how next-token prediction produces fluent answers that can still be wrong, and what that implies for grounding, determinism and testing.
- Estimate the tokens, cost and latency of a request before you build it, and say how wrong the estimate might be and why.
- Diagnose a failed run from its usage block and stop reason: truncation, context overflow, a buried fact, a stale model or a cache miss.
- Choose output limits, prompt layout and caching so a request fits an agreed latency and cost budget.
- Defend to a model-risk reviewer why "temperature 0" does not buy reproducibility, and what does.
Before you start
- This is the first concept of the programme. No earlier files are required.
- You have called an LLM API directly and read a JSON response.
- Basic probability: a distribution, sampling, expected value.
The short version
A large language model predicts a probability distribution over the next token; the serving system picks one, appends it, and repeats until something stops it. The model knows only its weights (frozen at a cutoff) and its context window (what you sent plus what it has written). Tokens are the unit of cost, latency and capacity, and tokenisers count the same text differently, especially outside English. Output tokens cost more and take longer than input, so answer length usually sets latency. Outputs vary even with sampling turned down, and long contexts lose facts in the middle. When an agent fails in front of a customer, the cause is almost always one of these mechanics, and the logs usually prove which.
- The model predicts plausible continuations, not verified facts. Fluency is not evidence.
- Tokens are the unit of everything. Count them per component, per language, per provider.
- Output length drives latency. Shorten the answer before you shop for a faster model.
- Non-determinism is a property of the system. Design validators and audit records around it.
- Always read the stop reason before you trust the output.
Pro lesson
The rest of this lesson is in Pro.
The mental model, worked example, field note, failure table, exercise and self-check, with the voice tutor.
Pro opens next
Stage 2 · One trustworthy model call- 3 lessons, with the voice tutor
- Halden week 2 · The dirty data pack lands
- The lab kit, to run Halden weeks on your machine
Also in Pro
- Every stage and specialisation, in full
- Every lab and every role project
- Halden weeks 2–10, on your record
- Every drill, live in the playground
- All solutions, plus the server judge
- Lab Assistant and every Teach with AI