Microsoft's ThinkingBox Reveals Alarming Reliability Gap in AI Agents
Microsoft has released an open-source sandbox framework called ThinkingBox to test AI agent reliability. The tool, developed by Principal Machine Learning Engineer Liang-Chun Tsai, takes a fundamentally different approach to evaluating AI agents.
Instead of grading them on what they say they did, ThinkingBox checks what they actually changed in a database. This approach is designed to reveal the 'discovery-reliability gap,' where AI agents appear to perform well initially but fail consistently when repeated trials are conducted.
The results of Microsoft's testing are sobering: even the best-performing model managed only a 25.25% pass^20 rate, meaning it succeeded across all 20 trials of the same task less than one-quarter of the time.
ThinkingBox comes paired with a benchmark called ThinkingBox-Bench and is now available on GitHub for developers to use and contribute to.