DeepMind ran a frontier model through a double-blind benchmark

Benchmark numbers carry a quiet problem: if a model has already seen the test questions during training, its score is inflated and no longer tells you much. The usual fix, holding out a private test set, forces a tradeoff, because the evaluator wants to keep the questions secret while the lab wants to keep the weights secret, and someone has to give. Google DeepMind says it ran the first double-blind evaluation of a proprietary, frontier-class model to remove that tradeoff. Using Google Cloud's Confidential Computing, specifically a Confidential Space, they built an isolated environment where the evaluators could not read the model weights and Google could not read the test prompts. A seven-step workflow cryptographically checks that both sides stayed blind.

The pilot ran on Gemini Flash Lite, with the Singapore AI Safety Institute, OpenMined, AVERI, and MLCommons taking part. Worth noting: the post does not publish any scores. The result being reported is the method itself, a way for an outside party to test a closed model on questions the lab never sees, so contamination stops being an easy way to game a benchmark.

Why it matters

If you pick models by their benchmark scores, contamination is a real reason those numbers can mislead you. This gives independent evaluators a path to test a closed model without the lab seeing the questions or the evaluator seeing the weights. If the approach spreads, published scores get harder to inflate, and third-party results become worth more.