Track 08 · Deployment & opsLesson 06 of 11
Canary releases and staged rollout
Move a workflow to an agent in steps, with a way back at each one.
- The FDE path · Stage 6 of 6
- 40 min read
- 60 min hands-on
- Pro
What you'll be able to do
01- Explain what "a release" is for an agent (image, model, prompts, tools, retrieval index, config) and why each part needs its own rollout and rollback path.
- Choose between rolling, blue/green, canary and shadow deployment for a given change, and between infrastructure-level and application-level (flag-based) rollout.
- Design a staged rollout by cohort and percentage, with sticky assignment, minimum samples per stage, bake times and automated rollback triggers drawn from error rate, latency, eval scores, guardrail hits and cost.
- Implement a deterministic, monotone cohort assignment and a staged rollout controller that advances or rolls back on canary analysis against a concurrent baseline.
- Defend a release plan, including kill switches and flag clean-up, to a change advisory board and a sceptical operations lead.
Before you start
- Online evals, A/B tests and flags: experiment design, randomisation units, guardrail metrics and significance. This concept assumes them.
- Treat prompts like releases: prompts as versioned release units resolved through labels.
- Regression tests and CI gates: nothing enters a rollout without passing the offline gate.
- CI/CD and the agent lifecycle and Containers and Kubernetes for agents: the signed artefact and the runtime it lands on.
- Retries, fallbacks and failing gracefully: the degraded modes a kill switch drops you into.
The short version
For an agent, a release can be a new image, model, prompt, tool, retrieval index or threshold, and several of those change without any deploy. So you separate deploying (new bits are running) from releasing (work is exposed to them), and you release in stages. Shadow the change on real inputs with write tools in dry-run. Then send a small, sticky, well-defined cohort to it: one work type, a few percent of shipments, never the tenants pinned by contract. Compare that cohort with a concurrent baseline on system, quality, safety and cost metrics. Advance only when a stage has enough sample and passes every gate; roll back automatically on any breach. Keep kill switches that drop the agent into a known-safe mode in seconds, and delete each release flag when the rollout ends.
- Deploy is not release. Flags and traffic routing let you ship dark and expose deliberately.
- A stage is a cohort and a percentage, with a minimum sample and a bake time. "1% for an hour" means nothing if 1% is twelve tasks.
- Compare against a concurrent baseline, not last week. Time is the biggest confounder.
- Rollback is pre-authorised and fast; advancing may need a human.
- Every flag has an owner and an expiry date.
Pro lesson
The rest of this lesson is in Pro.
The mental model, worked example, field note, failure table, exercise and self-check, with the voice tutor.
Pro opens this stage
Stage 6 · Ship, run and hand over- 8 lessons, with the voice tutor
- Lab · Check a trace export before you trust the dashboard, graded here
- Halden week 7 · The champion moves
- Halden week 8 · The portal goes down before cut-off
- Halden week 9 · Prices up, and a new rule
- Halden week 10 · A second site, and handover
- The lab kit, to run Halden weeks on your machine
Also in Pro
- Every stage and specialisation, in full
- Every lab and every role project
- Halden weeks 2–10, on your record
- Every drill, live in the playground
- All solutions, plus the server judge
- Lab Assistant and every Teach with AI