The article says threat modeling for AI applications has to change because AI systems do not behave like normal software. Traditional software is more predictable, but AI systems are probabilistic, can respond differently to the same input, and may treat text, images, or other outside content as instructions rather than just data. Microsoft says this creates new risks such as prompt injection, indirect prompt injection through external content, misuse of tools, hidden data leakage, and overconfident but wrong answers. The post also stresses that AI threat modeling should not focus only on databases and credentials: teams also need to protect things like user safety, trust, privacy, prompt integrity, and the integrity of agent actions.

Its main advice is to model the real system, not the simplified design version, and to build protections into the architecture from the start. Microsoft recommends mapping where untrusted data enters the system, understanding how prompts, memory, retrieved context, and tools are connected, and treating those “in-between” handoff points as major security boundaries. For defense, it highlights practical controls like separating system instructions from untrusted content, limiting tool permissions, using allow lists, requiring human approval for high-risk actions, validating outputs before they leave the system, and designing strong logging and response paths. The overall message is that AI threat modeling is not a one-time checklist—it should be an ongoing discipline that helps teams reduce blast radius, detect failures early, and keep AI systems trustworthy as they evolve.

‍

Source: https://www.microsoft.com/en-us/security/blog/2026/02/26/threat-modeling-ai-applications/