LLM judges and rubrics you can trust

An LLM judge is a model that grades another model's output against a rubric. It is how you grade faithfulness, tone and adequacy across thousands of cases when code cannot decide and SMEs cannot read them all. It is also an instrument that can be biased, lenient or inconsistent, so it is not valid until you have compared it with human labels. Use judges only where code cannot grade. Write rubrics as small, binary, decomposed criteria with definitions and examples, not as a 1–10 scale. Ask for reasoning before the verdict, constrain the output to a schema, and treat the graded text as data. Calibrate on a held-out set of SME labels and report the confusion matrix and Cohen's kappa, not raw agreement. Check position bias in pairwise judges by swapping the order. Version the judge like any release, re-calibrate when anything in it changes, and never change the judge and the system in the same comparison.

• A judge is an instrument; calibrate it before you trust it.
• Binary, decomposed criteria beat unanchored numeric scales for agreement, calibration and action.
• Kappa, not raw agreement. On an imbalanced set, a judge that always says "pass" can agree 95% of the time and be worthless.
• Known biases are predictable. Position, verbosity and self-preference have documented mitigations.
• Judges drift when their model or prompt changes. Pin, version, re-calibrate, and run canary items every time.