Track 05 · EvalsLesson 09 of 10

Acceptance criteria and sign-off

Agree in writing what working means before anything is built.

What you'll be able to do

What you'll be able to do

01
  • Write an acceptance-criteria document before the build: behavioural criteria in Given/When/Then form, quality criteria as metric thresholds on named eval sets, and non-functional criteria for latency, cost, security and privacy.
  • Link every criterion to an eval set version, a baseline and a measurement method, and state thresholds as confidence bounds rather than point estimates.
  • Plan user acceptance testing (UAT) with the customer's real users, with entry and exit criteria and a defect triage scheme.
  • Assemble a sign-off pack: test report, known limitations, residual risk register and evidence pack, and identify who must sign at each phase gate.
  • Handle scope changes through change control, and convert "it feels wrong" feedback into test cases.
  • Defend a "not ready" verdict to a sponsor who wants a date.
Before you start

Before you start

The short version

The short version

"Working" is a claim, and customers sign claims only when they are written down and measurable. Before you build, agree a short acceptance-criteria document with the customer: what is in and out of scope; behavioural criteria written as Given/When/Then scenarios for things that must always or never happen (escalate above a limit, never act without approval); quality criteria as thresholds on named, versioned eval sets, with baselines and minimum sample sizes; and non-functional criteria for latency, cost, availability, security and privacy. State quality thresholds as confidence bounds, so "96% on 320 cases" is not mistaken for "at least 95%". Plan UAT with the customer's real users and a defect scheme agreed in advance. At each phase gate (pilot, limited release, general availability), bring a sign-off pack: results per criterion with evidence links, known limitations, a residual risk register with named owners, and the release identifiers of what was tested. When the customer changes their mind, run it through change control: a new criterion needs new eval cases, a new baseline and possibly a new date. When someone says "it feels wrong", ask for examples and turn them into cases. Name who signs (business owner, risk, security, privacy, operations) before you need their signatures.

  • Write it down first. Criteria agreed after results are seen are negotiated, not measured.
  • Every criterion has a measurement method, a data set, a threshold and an owner.
  • Thresholds are lower bounds, not point estimates. Small eval sets cannot prove high thresholds.
  • Sign-off is per release and per gate. It covers a specific prompt, model and configuration, not "the assistant".
  • Change is normal; unrecorded change is the problem.

Pro lesson

The rest of this lesson is in Pro.

The mental model, worked example, field note, failure table, exercise and self-check, with the voice tutor.

Pro opens this stage

Stage 5 · Prove it and pass review
  • 7 lessons, with the voice tutor
  • Lab · Clean a golden set before you trust it, graded here
  • Halden week 4 · First slice due
  • Halden week 6 · Instructions hidden in an invoice
  • The lab kit, to run Halden weeks on your machine

Also in Pro

  • Every stage and specialisation, in full
  • Every lab and every role project
  • Halden weeks 2–10, on your record
  • Every drill, live in the playground
  • All solutions, plus the server judge
  • Lab Assistant and every Teach with AI