Regression tests and CI gates

Every change to an LLM application can change its behaviour: a prompt edit, a new tool description, a retrieval setting, a library upgrade, and most of all a model change. A regression gate is the automated check that stands between those changes and production. It runs a fixed suite (golden cases, canaries, must-pass safety cases and every past failure) against the candidate and the current baseline, compares them case by case, and blocks the merge if quality drops beyond agreed limits, overall or in any slice, or if a single must-pass case fails. Because models are non-deterministic, the gate runs cases several times, treats unstable cases as flaky rather than as pass or fail, and uses a paired test to tell real regressions from noise. It runs fast and cheap on every push using recorded responses, and live, under a budget, before merge. When a provider deprecates a model, the replacement goes through exactly the same gate.

Key takeaways:

• The suite grows from incidents: every production failure becomes a permanent test case.
• Gate per slice, not only overall. Averages hide the language or segment that broke.
• One must-pass failure blocks the change, whatever the overall score.
• Run each case several times; quarantine and fix flaky cases instead of rerunning until green.
• Compare candidate with baseline on the same cases, paired. Unpaired averages waste statistical power.