Golden datasets and the metrics that matter

A golden set is a curated, labelled, versioned sample of the work the system must do, with a known procedure for each label and a named owner. It is the reference every eval runs against, so its flaws become your conclusions. Build it mostly from real production traffic, masked, and stratified so rare but important cases are present in useful numbers. Add SME-written edge cases and past incidents, and use synthetic inputs carefully and labelled as such. Size it from the confidence interval you need, not from a round number: 20 cases can tell you something is broken, but it takes hundreds to show something meets a threshold. Write labelling guidelines, double-label a share, and keep the set away from everything that shapes the system: prompt examples, judge examples and fine-tuning data. Version it like code. Choose metrics that match the task, and do not trust n-gram overlap metrics for open-ended text.

• Representative first, then enriched. Sample from production by stratum, then add edge and adversarial cases in separate suites.
• Report intervals, not just rates. Use the Wilson interval; re-weight stratified samples to the production mix for headline numbers.
• Guidelines make labels reproducible. Double-label, measure agreement, adjudicate, record rationale.
• Leakage makes scores meaningless. Keep dev and test splits, deduplicate, and never put test cases in prompts.
• Datasets are versioned artefacts with owners. Every result records the dataset version that produced it.