Threat modelling an AI system

A threat model answers four questions: what are we building, what can go wrong, what are we doing about it, and did we do a good enough job. For an agent, you start with a data-flow diagram of the real deployment: users, UIs, the agent runtime, the model provider, retrieval stores, memory, tools and the MCP servers behind them, systems of record, logs and traces, with a trust boundary wherever data crosses between parties or privilege levels. The agent-specific insight is that the model is a confused deputy: it holds your privileges and follows instructions from anything it reads. You then walk each element with STRIDE, build attack trees for the worst outcomes, and name each threat in the vocabulary the security team already uses: OWASP LLM01–LLM10 (with the edition year, because the 2026 edition renumbers most of the 2025 list), OWASP ASI01–ASI10 for agentic applications (2026) and MITRE ATLAS technique IDs. Each threat gets an inherent rating, the controls that address it (with links to evidence), a residual rating and an owner. Residual risk that remains is written down and accepted by a named person with an expiry date. The document is versioned with the code and revisited whenever a tool, data source, model or channel changes.

Key takeaways:

• Model the system as deployed, not as on the slide. Missing elements are where the findings hide.
• Treat the model as an untrusted process inside your boundary: it can be talked into using every privilege you give it.
• OWASP gives you names the security team recognises; ATLAS gives attacker technique IDs for red teams and detection engineers.
• Every control in the register needs evidence (a test, a scan, a red-team result). A control without evidence is a claim.
• A threat model nobody updates is wrong within a quarter. Make change triggers part of the delivery process.