Track 08 · Deployment & opsLesson 08 of 11
Dashboards the sponsor actually reads
Show business value, not spans and percentiles.
- Lesson library · reference
- 30 min read
- 60 min hands-on
- Pro
What you'll be able to do
01- Explain which question each stakeholder (exec sponsor, ops lead, risk and compliance, engineering) brings to a dashboard, and why one view cannot answer all four.
- Define outcome metrics as contracts: numerator, denominator, exclusions, time attribution, maturity window and named owner.
- Implement a weekly metrics pack that reports rates with Wilson intervals and sample sizes, week-over-week signals, segment breakdowns and release annotations.
- Diagnose vanity metrics, Goodhart effects and aggregates that hide a segment-level regression.
- Design a dashboard layout (top line, trends, drill-down) and a weekly business review the customer keeps running after you leave.
Before you start
- Tracing agents with OpenTelemetry: the telemetry summarised here, and the exception ID that joins a trace to a business outcome.
- Querying a month of traces: pulling trace and eval data into tables you can aggregate.
- Online evals, A/B tests and flags: guardrail metrics and significance, reused for week-over-week comparison.
- Human review and feedback loops: where QA verdicts come from.
- SLOs and latency budgets: the targets the engineering view reports against.
- Baselines and ROI you can defend: the pre-build baseline behind "hours saved".
The short version
A business dashboard translates telemetry into the few measures the customer agreed would prove value: how much work the agent handles, how much time it saves, how good the output is, what each resolved case costs, and whether people use it. Each measure is a contract (numerator, denominator, exclusions, maturity, owner) written before any number appears. Different audiences ask different questions, so you build a view for each on one shared metric layer. Every rate shows its sample size and an interval, every trend carries release and incident annotations, and every top-line number can be segmented, because averages hide regressions. The dashboard earns its keep in a weekly review where an owner explains each movement.
- Outcomes at the top, system metrics underneath. p95 latency belongs to engineering; "drafts ready within 5 minutes" belongs to the sponsor.
- A metric is a definition before it is a number.
- Show n and an interval. A 7-point jump on 46 QA samples is usually noise.
- Segment what matters. A flat top line can hide one language improving and another breaking.
- Annotate changes, or every movement becomes an argument.
Pro lesson
The rest of this lesson is in Pro.
The mental model, worked example, field note, failure table, exercise and self-check, with the voice tutor.
- Every stage and specialisation, in full
- Every lab and every role project
- Halden weeks 2–10, on your record
- Every drill, live in the playground
- All solutions, plus the server judge
- Lab Assistant and every Teach with AI