Catching drift before the customer does

A signed, working agent will get worse without anyone changing its code. Users start asking about a product launched after your golden set was frozen. A policy changes, so last quarter's correct answer is now wrong. The provider updates or retires the model. Someone replaces a document in the knowledge base. A prompt is edited outside the registry. Users trust the system more and check it less. None of this throws an error; the agent keeps answering. You catch it by monitoring distributions and quality over time: the mix of intents and languages (PSI is a simple, explainable measure for categorical mixes), embedding-based shift and new-cluster rates for free text, online judge scores, guardrail hit rates, feedback, escalations and outcomes. Put quality metrics on control charts rather than fixed thresholds, so alerts fire on real shifts and not on daily noise, and send one notification per episode rather than one per day. Re-run the golden set on a schedule even when nothing has changed, refresh it from production with current labels, and keep a model deprecation playbook tied to the providers' notice periods. Every confirmed drift ends as new golden cases and a gated fix.

• Drift is the default. Plan to detect it; do not hope to avoid it.
• Watch inputs, outputs, outcomes and the system, because each kind of drift shows up in a different place first.
• Use statistics that respect noise. Control charts and minimum sample sizes beat hand-picked fixed thresholds.
• Re-evaluate on a schedule, not only on change. Upstream changes do not open a pull request in your repo.
• A retirement notice starts a project, not a config change. Budget time for evaluation and re-approval.