Microsoft's `run-assert-eval` Cuts AI Agent Data Leaks in Half

Microsoft has released an open-source tool called `run-assert-eval` to automate security for AI agents. The tool, released on September 24, 2026, chains four stages into one repeatable workflow: identifying vulnerabilities, converting risks into measurable tests, generating a runtime security policy, and verifying that the policy works without breaking anything else.

The tool is designed to address the problem of written AI safety policies not being enforced, and standard code review missing agent-specific failures like cross-account data leakage. The demo shows a billing support agent exposing other customers' data in 30% of conversations before `run-assert-eval` generated and applied a policy blocking calls with mismatched account identifiers, reducing the rate to 5.9%.

The tool builds on existing tools from Microsoft, including ASSERT and ACS, which were released as separate tools in June 2026. It also draws on PyRIT (Python Risk Identification Tool), which was integrated into Azure in spring 2025 for model-level red teaming. The main open-source alternative remains NVIDIA's Garak framework, but `run-assert-eval` targets the gap between threat modeling and runtime enforcement.