Online evals, A/B tests and flags

Offline evals tell you whether a change is safe to try. Only real users tell you whether it is better. Golden sets are frozen samples of yesterday's traffic, judges are proxies, and users change their behaviour when the assistant changes. So after a change passes its CI gate, you measure it on live traffic. Start with shadow mode when the risk is high (the candidate runs on real inputs but nobody sees its output), then run a randomised controlled experiment when you need a causal answer. A good experiment is designed before it starts: one primary metric tied to the business outcome, guardrail metrics for safety, latency and cost that can veto a launch, a randomisation unit that matches how users interact, sticky assignment, a sample size computed from a minimum detectable effect, and a fixed duration or a sequential method that tolerates peeking. Feature flags carry the variant assignment in code, and every trace records which variant served it. Online judges score a sample of live traffic, and that sample is personal data, so redact and govern it.

• Offline gates decide what is safe to test. Online experiments decide what ships.
• Pick the metric, the unit, the sample size and the stop rule before day one. Changing them after you see data invalidates the test.
• Guardrails can veto a winning primary metric. A faster resolution that raises policy violations is not a win.
• Check the split before you read the result. A sample ratio mismatch means the data is broken, not that the treatment worked.
• Sampled production traffic is regulated data. Redaction, retention and residency apply to judges and reviewers too.