Track 09 · Systems & unit costLesson 10 of 11
Scaling for the week after launch
Plan for traffic that looks nothing like the pilot.
- Lesson library · reference
- 30 min read
- 75 min hands-on
- Halden Freight
- Pro
What you'll be able to do
01- You can explain why agent workloads scale differently from request/response services, and name the constraint that usually binds first.
- You can turn a volume forecast into concurrency, worker, RPM and TPM numbers with Little's law, counting tokens the way your provider does.
- You can design load levelling with bounded queues, deadlines, backpressure and per-tenant fairness.
- You can choose between on-demand quota, priority tiers and provisioned throughput on Bedrock, Azure and Vertex.
- You can plan and cost a load test that finds the first bottleneck before a peak, and defend a degradation order.
Before you start
- Containers and Kubernetes for agents (pods, KEDA, graceful drain)
- SLOs and latency budgets
- Retries, fallbacks and failing gracefully
- A gateway for bursty agent traffic and Routing models to fund the rollout
- Cost per resolved case
- Background: arrival rate, service time and utilisation as queueing terms.
The short version
An agent run is long, holds a worker for its whole duration, and spends most of that time waiting on a model whose throughput you rent by the token. Capacity is therefore concurrency (arrival rate × run duration, by Little's law) set against a provider token quota and several downstream API budgets, any of which can bind first. Plan from a forecast, put a bounded queue and a token-aware limiter between demand and the provider, autoscale workers on queue depth up to the quota-derived ceiling, reserve capacity for known peaks, and agree in advance what sheds first.
- L = λW turns a forecast into a concurrency number. Size everything else from it.
- The provider quota, not your cluster, is usually the ceiling. Past it, more workers make latency worse.
- Know which tokens your provider counts and when:
max_tokensat admission, output multipliers, cache reads. - Bounded queues with deadlines turn overload into deferred work instead of timeouts.
- Load test with stubs for the platform and a short real-model run for the provider, in an isolated quota.
Pro lesson
The rest of this lesson is in Pro.
The mental model, worked example, field note, failure table, exercise and self-check, with the voice tutor.
- Every stage and specialisation, in full
- Every lab and every role project
- Halden weeks 2–10, on your record
- Every drill, live in the playground
- All solutions, plus the server judge
- Lab Assistant and every Teach with AI