DeepMind runs a frontier model evaluation where neither side can peek

External model evaluations have always forced an awkward choice. Either the evaluator hands over its test questions, which risks the model owner seeing them in advance and inflating scores, or the owner hands over model weights, which risks losing valuable intellectual property. Google DeepMind says it has run the first double-blind evaluation of a proprietary frontier-class model that avoids both. The test prompts stay hidden from Google, the model weights stay hidden from the evaluator, and each side gets cryptographic proof that its data stayed private.

The mechanism is Google Cloud's Confidential Space, which builds a hardware-backed enclave around a GPU where the evaluation runs. Neither party can read the other's inputs, and a seven-step workflow moves data between the model owner and the evaluator without either one gaining access to the other's secret. The pilot ran a Gemini Flash Lite model against confidential benchmarks, with partners including the Singapore AI Safety Institute, OpenMined, AVERI, and MLCommons. As DeepMind puts it, if a model can peek at the questions ahead of time, the score becomes meaningless, and that trust matters most for sensitive areas like cybersecurity and government use.

Why it matters

If you rely on third-party benchmark numbers to judge a model, contamination is the quiet failure that makes them lie. A setup where the lab provably cannot see the questions gives those scores real weight, and it lets safety institutes test closed models without demanding the weights.