From requirements to eval criteria

An eval is only as good as the definition of "good" behind it. Customers state requirements as outcomes and feelings ("resolve disputes right first time", "it feels wrong sometimes"). You turn each one into observable behaviours, each behaviour into a binary criterion that can be graded case by case, and each criterion into a measurement with a grader, a threshold, a severity and an owner. Prefer code checks, use model judges where judgement is needed, and use humans where stakes or ambiguity demand it and to calibrate the judges. Set thresholds against the measured human baseline and the cost of each kind of error, not against round numbers. Write the spec before you build or change anything, version it, and get the customer to sign it. That signed spec is what turns a pilot review from an argument about anecdotes into a decision about numbers.

• A criterion must be observable, gradable per case and traceable to a requirement. If it fails any of the three, it is not a criterion yet.
• Push each criterion down the grader ladder: code if possible, a calibrated model if not, humans for calibration and the highest-stakes calls.
• "Must never" requirements are hard gates, and a zero in the eval does not prove a zero in production.
• Thresholds come from baselines, business cases and risk appetite, reported with the uncertainty the sample size allows.
• The eval spec is a customer artefact. Engineering maintains it; the business signs it.