Microsoft Insights in Foundry Simplifies AI Agent Error Detection

Microsoft is introducing a new feature in its Foundry AI development and governance platform designed to streamline the process of identifying and correcting errors made by AI agents. The feature, called Insights in Foundry, analyzes traces of agent activity to pinpoint the likely causes of mistakes and suggests ways to fix them. It identifies recurring and previously unknown behaviors, provides evidence-based explanations, and recommends actions for developers to resolve issues.

Insights in Foundry builds on existing capabilities like tracing, monitoring, and evaluation, but goes further by analyzing large volumes of traces stored in Azure Monitor Application Insights. This allows it to detect patterns that would be difficult to spot by examining individual traces. The patterns help developers investigate problems related to tool calls, context handling, output quality, reliability, latency, or unnecessary token use.

When Insights confirms a finding, developers can take several steps to address it. These include making targeted changes to instructions or workflows, adding evaluation coverage to create or update datasets, routing issues to their owners for fixes, or exploring alternatives using optimization workflows like Agent Optimizer. However, Insights does not implement changes automatically; all updates require human review and follow standard development processes.

Microsoft provided an example to illustrate how Insights works. In a scenario involving an AI agent handling customer order support, Insights could analyze traces to reveal a pattern where an order-status lookup failure leads to an incorrect claim that an order was delivered. Developers can then inspect the requests, tool calls, and responses to determine the root cause and test potential fixes before deploying them.