Microsoft's AI Testing Agent Cuts Failures by 63%

Microsoft's new testing agent aims to address the growing concern over AI-generated code. The company claims its open-source tool can improve test generation and reduce failures by up to 63%. This is a significant improvement over existing solutions, such as GitHub Copilot.

The agent uses a combination of research and iterative planning to generate unit tests for specific codebases. It avoids shallow testing, which only checks if a result isn't null, and instead focuses on more comprehensive coverage. The tool also runs mutation testing to verify that the generated tests can catch injected bugs.

Microsoft benchmarked its agent against stock GitHub Copilot using 152 tasks pulled from real repositories. The results showed that the specialized agent completed 92.1% of tasks compared to 78.9% for stock Copilot. This means that developers who use Microsoft's agent are less likely to ship untested AI-generated code.

According to Mitch Ashley, VP and practice lead for software lifecycle engineering and AI-native software engineering at The Futurum Group, the real story is in the method, not the model. He emphasizes that coverage numbers have been lying to teams for years, and Microsoft's agent runs mutation testing on its own output to confirm the tests actually notice.