Topic 05 of 10
Evals
Proof before claims: turning requirements into eval criteria, golden sets, LLM judges, CI gates and the acceptance tests a client signs off.
- 10 lessons
- 6.8 h reading
- 10.5 h hands-on
- 5 on the FDE path
Start lesson 01
Progress0 / 10
Lessons
- 01From requirements to eval criteriaPath · stage 5Turn what the customer asked for into checks a test can pass or fail. Pro
- 02Golden datasets and the metrics that matterPath · stage 5 LabWrite the test set down, pin it to numbers, and improve against it. Pro
- 03LLM judges and rubrics you can trustPath · stage 5Scale review with a judge model, calibrated against people. Pro
- 04Evaluating each step, not just the answerFind where a multi-step agent goes wrong, tool call by tool call. Pro
- 05Regression tests and CI gatesPath · stage 5Block the prompt or model change that quietly breaks a live workflow. Pro
- 06Online evals, A/B tests and flagsMeasure what real users do, safely, once offline evals say yes. Pro
- 07Human review and feedback loopsRead outputs on a cadence, label them well, and feed the fixes back. Pro
- 08Querying a month of tracesSpecialisationDebug production at scale with queries instead of scrolling logs. Pro
- 09Acceptance criteria and sign-offPath · stage 5Agree in writing what working means before anything is built. Pro
- 10Catching drift before the customer doesSpot slow quality decline as data, usage and models shift underneath. Pro