Google DeepMind Tackles Benchmark Contamination with Double-Blind Evaluation

Google DeepMind has launched a pilot project to address the issue of benchmark contamination in AI model evaluation. The company is using a cryptographic method called Confidential Space from Google Cloud's confidential computing portfolio to ensure that external tests stay locked in a 'box' and can't be used by the model provider to optimize itself specifically for the test.

The double-blind evaluation aims to eliminate the tradeoff between handing over test prompts, which lets the model provider see the questions in advance, or handing over model weights, which risks the intellectual property of the provider. This issue was highlighted by a recent example, where the delayed evaluation for Anthropic's Fable 5 was due to the company's 30-day data retention policy.

Confidential Space uses cryptographic verification to ensure that both the external test data and the model stay private to their respective owners. The evaluator never sees the Gemini weights, and Google never sees the test prompts. This method is particularly important for sensitive areas such as cybersecurity and government agency tests, where independent organizations can rigorously test advanced models without compromising data sovereignty or security.

Google hopes that this effort will set a new standard for model oversight and help build more reliable and widely trusted AI systems.