Track 06 · Security & risk reviewLesson 03 of 10
Automated red-teaming
Attack your own agent at scale before an auditor or a journalist does.
- Lesson library · reference
- 40 min read
- 75 min hands-on
- Pro
What you'll be able to do
01By the end of this file you can:
- Write rules of engagement for a red-team campaign against a customer's agent: scope, data, budgets, stop conditions and evidence handling.
- Design an attack plan that maps each attack family to an OWASP Top 10 for LLM Applications category and to a concrete, checkable harm.
- Explain how attacker-LLM methods (PAIR, TAP), seed mutation, multi-turn escalation (Crescendo) and indirect, tool-borne attacks differ, and choose between them for a given target.
- Implement a scoring layer that combines deterministic canaries and state checks with LLM judges, and report attack success rate with a confidence interval.
- Convert confirmed findings into regression cases and a severity-ranked report a risk committee can act on.
Before you start
- Threat modelling an AI system: the threat model is the input to your attack plan. Read it first even though it is numbered later.
- Defending against prompt injection: the controls you are about to attack.
- LLM judges and rubrics you can trust and Regression tests and CI gates.
- Background: what a binomial proportion is, and why 0 out of 20 is not the same as 0%.
The short version
Red-teaming is authorised attack against your own system to measure how often, and how badly, it can be made to misbehave. Automation turns a few human attacks into thousands of variants: threat-model seeds, mutated by deterministic transforms and attacker LLMs, delivered directly, through documents and tools, and across turns. Each attempt is scored by a deterministic check where possible (did the canary leak, was the tool called with another member's ID) and by an LLM judge only where necessary. The output is an attack success rate per category with an honest interval, reproducible findings ranked by severity, and a regression suite that keeps them fixed.
- Scope and rules of engagement come first. An unscoped red team is an incident.
- Report attack success rate (ASR) with a Wilson interval and the attempt count. Zero successes in 24 tries still allows a true rate near 14%.
- Every confirmed finding becomes a regression case, paired with a benign case so the fix does not create over-refusal.
- Automated ASR is a lower bound on what a motivated human achieves. Keep humans on novel attack classes.
Pro lesson
The rest of this lesson is in Pro.
The mental model, worked example, field note, failure table, exercise and self-check, with the voice tutor.
- Every stage and specialisation, in full
- Every lab and every role project
- Halden weeks 2–10, on your record
- Every drill, live in the playground
- All solutions, plus the server judge
- Lab Assistant and every Teach with AI