Track 05 · EvalsLesson 08 of 10
Querying a month of traces
Debug production at scale with queries instead of scrolling logs.
- Specialisation · Data & AI platform
- 40 min read
- 75 min hands-on
- Pro
What you'll be able to do
01- Explain the trace data model (trace, span or observation, generation, session, user, scores, metadata) as exposed by platforms such as Langfuse and Arize Phoenix.
- Design a warehouse layout for exported traces, scores and business outcomes, with the right grain and join keys.
- Implement SQL for failure analysis: error rate by tool, latency percentiles, cost per session, top failing intents and drill-downs to the retrieved document.
- Diagnose a quality regression by slicing on release, prompt version and model, and separate a release effect from an upstream incident happening at the same time.
- Validate an eval score against business outcomes by joining traces with case data from the system of record.
- Choose sampling, retention and PII controls for trace data that a privacy officer will accept.
Before you start
- Evaluating each step, not just the answer: component-level scores are what you will slice here.
- Online evals, A/B tests and flags: variants and online judge scores live on traces.
- Human review and feedback loops: query results decide what reviewers see.
- Treat prompts like releases: the release hash on every trace is what makes slicing by version possible.
- Working SQL: joins,
GROUP BY, CTEs and window functions.
Instrumentation itself (OpenTelemetry, GenAI semantic conventions, span design) is taught in Tracing agents with OpenTelemetry. This concept assumes traces are already arriving through a platform SDK and teaches you to query them.
The short version
A production agent emits a trace per request: a tree of spans (Langfuse calls them observations) for the agent loop, each retrieval, each tool call and each model call, with metadata such as intent, release and user, and scores attached by judges, users and reviewers. A month of that is millions of rows. Scrolling a trace viewer cannot tell you which tool fails most, whether release 1.4.0 is slower, or whether low judge scores predict reopened cases. SQL can. Export traces and scores on a schedule into a warehouse (or query them in the platform), keep one row per span with trace-level attributes copied onto every row, and join in business outcomes from the customer's system of record. Then the standard questions are a handful of queries: error rate by tool, latency percentiles by release, cost per session, failure rate by intent, drill-down to the document or tool behind the failures, and judge score against outcome. Slice by release and prompt version, but watch for confounding with time. Mask PII before traces are stored, set retention deliberately, and give query access on the same terms as the production data it came from.
- Grain first. Know whether a row is a span, a trace, a session or a score before you write
COUNT(*). - Every trace carries release, prompt version and model. Without them you cannot slice, and without slicing you cannot debug.
- Join to outcomes. A judge score that does not predict reopened cases or complaints is not measuring what the business cares about.
- Before and after is not an experiment. Releases coincide with incidents, month-end and product launches; separate them before blaming the release.
- Trace data is customer data. Mask at source, minimise exports, and set retention.
Pro lesson
The rest of this lesson is in Pro.
The mental model, worked example, field note, failure table, exercise and self-check, with the voice tutor.
- Every stage and specialisation, in full
- Every lab and every role project
- Halden weeks 2–10, on your record
- Every drill, live in the playground
- All solutions, plus the server judge
- Lab Assistant and every Teach with AI