Google Unveils Double-Blind Evaluation System for AI Model Testing

Google DeepMind has developed a system to test its Gemini AI model without revealing its questions or weights. The double-blind evaluation, made possible by Google Cloud's Confidential Space and NVIDIA's H100 Confidential GPU, keeps both parties' data private during the testing process.

The pilot test used MLCommons and Singapore AI Safety Institute benchmarks, but instead of showcasing a new Gemini score, Google focused on the innovative testing method. The company cited research showing that benchmark leakage can inflate scores, particularly for larger models, and contaminate results.

To mitigate this issue, the system uses encrypted connections to transmit data into an enclave where the evaluation takes place. The model weights are stored in hardware-encrypted GPU memory, while the evaluation prompts are kept in encrypted host memory. After testing, only the permitted results are given to the evaluator.