Track 09 · Systems & unit costLesson 03 of 11
Voice agents and conversation design
The voice stack, turn-taking and channels, from speech in to speech out.
- Specialisation · Customer experience & voice
- 30 min read
- 75 min hands-on
- Pro
What you'll be able to do
01- You can choose between cascaded STT → LLM → TTS and speech-to-speech for a regulated use case, and defend it.
- You can decompose a voice turn into stages and build a per-stage latency budget that you then verify end to end.
- You can design conversations for the ear, including confirmation of spoken identifiers and a DTMF fallback.
- You can map where PHI lives in a voice system and explain the controls to a CISO.
- You can define a voice eval plan: WER, entity accuracy, task success, latency percentiles.
Before you start
- Cost per resolved case (taught before this file in the week's sequence).
- SLOs and latency budgets.
- Handing off to a person, well.
- Finding and masking personal data.
The short version
A voice agent is a chat agent that must decide when the caller has finished, start answering within about a second, and say things that make sense heard once with no screen. You either chain speech-to-text, an LLM and text-to-speech (cascaded) or use a speech-to-speech model. Cascaded gives control and a text audit trail; speech-to-speech gives lower latency with less visibility. Most latency comes from endpointing and tool calls, and most failures from conversation design.
- Budget latency per stage, then measure the whole; streaming and better endpointing usually beat a faster model.
- Spoken identifiers need a deterministic confirmation policy, not a better prompt.
- Audio and transcripts are PHI for Carevia; every vendor in the audio path needs a BAA.
- Evaluate entity accuracy and task success per language and accent, not aggregate WER.
Practise this in the lab
Pro lesson
The rest of this lesson is in Pro.
The mental model, worked example, field note, failure table, exercise and self-check, with the voice tutor.
- Every stage and specialisation, in full
- Every lab and every role project
- Halden weeks 2–10, on your record
- Every drill, live in the playground
- All solutions, plus the server judge
- Lab Assistant and every Teach with AI