Tracing agents with OpenTelemetry

A trace is the record of one unit of work as it moves through your system: one tree of timed spans, each with a name, a parent, a status and attributes. For an agent, the root span is the agent run, and its children are the model calls, tool calls, retrievals, subagent runs and queue hops that made it up. OpenTelemetry gives you a vendor-neutral way to produce this: an API your code calls, an SDK that samples, processes and exports spans, the OTLP protocol, and a Collector that redacts, samples and routes before anything reaches a backend. W3C Trace Context carries the trace across process boundaries; if one hop drops it, the trace breaks there and the slow part becomes invisible. The OpenTelemetry GenAI semantic conventions name the spans and attributes (invoke_agent, chat, execute_tool, gen_ai.usage.input_tokens), but they are still in Development status, so pin and verify. Prompts and completions are sensitive: leave content capture off by default, redact at source, redact again in the Collector, and keep traces joinable to evals and business outcomes through IDs rather than content.

• One unit of work, one connected trace. Propagate context across every hop, including queues.
• Span names are low-cardinality; attributes carry the detail. chat <model>, not chat for case 118734.
• The GenAI conventions are useful and unstable. Status Development as of September 2026; check before you build dashboards on attribute names.
• Content capture is opt-in, and redaction happens twice. At source, and in the Collector.
• Sample at the tail for agents. Keep every error, handoff and slow run; sample the healthy majority.