Track 05 · EvalsLesson 09 of 10
Acceptance criteria and sign-off
Agree in writing what working means before anything is built.
- The FDE path · Stage 5 of 6
- 40 min read
- 60 min hands-on
- Pro
What you'll be able to do
01- Write an acceptance-criteria document before the build: behavioural criteria in Given/When/Then form, quality criteria as metric thresholds on named eval sets, and non-functional criteria for latency, cost, security and privacy.
- Link every criterion to an eval set version, a baseline and a measurement method, and state thresholds as confidence bounds rather than point estimates.
- Plan user acceptance testing (UAT) with the customer's real users, with entry and exit criteria and a defect triage scheme.
- Assemble a sign-off pack: test report, known limitations, residual risk register and evidence pack, and identify who must sign at each phase gate.
- Handle scope changes through change control, and convert "it feels wrong" feedback into test cases.
- Defend a "not ready" verdict to a sponsor who wants a date.
Before you start
- From requirements to eval criteria: this concept turns those criteria into a signed contract.
- Golden datasets and the metrics that matter: criteria reference versioned eval sets.
- LLM judges and rubrics you can trust: judge-scored criteria are only as good as judge calibration.
- Regression tests and CI gates: acceptance criteria become the release gate.
- Human review and feedback loops: human labels with measured agreement back the quality criteria.
- Baselines and ROI you can defend: the business baseline that criteria are compared against. (Week 10 is taught last but applied from week 1.)
The short version
"Working" is a claim, and customers sign claims only when they are written down and measurable. Before you build, agree a short acceptance-criteria document with the customer: what is in and out of scope; behavioural criteria written as Given/When/Then scenarios for things that must always or never happen (escalate above a limit, never act without approval); quality criteria as thresholds on named, versioned eval sets, with baselines and minimum sample sizes; and non-functional criteria for latency, cost, availability, security and privacy. State quality thresholds as confidence bounds, so "96% on 320 cases" is not mistaken for "at least 95%". Plan UAT with the customer's real users and a defect scheme agreed in advance. At each phase gate (pilot, limited release, general availability), bring a sign-off pack: results per criterion with evidence links, known limitations, a residual risk register with named owners, and the release identifiers of what was tested. When the customer changes their mind, run it through change control: a new criterion needs new eval cases, a new baseline and possibly a new date. When someone says "it feels wrong", ask for examples and turn them into cases. Name who signs (business owner, risk, security, privacy, operations) before you need their signatures.
- Write it down first. Criteria agreed after results are seen are negotiated, not measured.
- Every criterion has a measurement method, a data set, a threshold and an owner.
- Thresholds are lower bounds, not point estimates. Small eval sets cannot prove high thresholds.
- Sign-off is per release and per gate. It covers a specific prompt, model and configuration, not "the assistant".
- Change is normal; unrecorded change is the problem.
Pro lesson
The rest of this lesson is in Pro.
The mental model, worked example, field note, failure table, exercise and self-check, with the voice tutor.
Pro opens this stage
Stage 5 · Prove it and pass review- 7 lessons, with the voice tutor
- Lab · Clean a golden set before you trust it, graded here
- Halden week 4 · First slice due
- Halden week 6 · Instructions hidden in an invoice
- The lab kit, to run Halden weeks on your machine
Also in Pro
- Every stage and specialisation, in full
- Every lab and every role project
- Halden weeks 2–10, on your record
- Every drill, live in the playground
- All solutions, plus the server judge
- Lab Assistant and every Teach with AI