Ai2's BenchMIRT digs into what an LLM benchmark actually measures

The Allen Institute for AI released BenchMIRT, a tool that looks inside a benchmark question by question to work out which abilities actually drive a model's score, instead of treating the benchmark as a single number. It uses multidimensional item response theory, a method borrowed from educational testing, and the team fit it on 100 models across 16 benchmarks and more than 34,000 questions. Without being given any labels, it kept recovering two main dimensions across runs: safety and general reasoning.

Some of the results cut against what a benchmark claims to test. BBQ, built to measure social bias, lined up more with general reasoning than with safety. On WMDP, which probes dangerous knowledge, stronger reasoning went with lower scores, because refusing to answer is the desired behavior. The Ai2 team also found you can keep just 10% of a benchmark's questions and still preserve most of the distinctions between benchmarks, and predict held-out answers with 79% accuracy against a 70% baseline. Their write-up walks through the method.

Why it matters

If you choose models or ship safety claims based on benchmark scores, this is a caution that a benchmark's name may not match what it rewards. The pruning result is also practical: you can build much cheaper evals by keeping the questions that carry the signal.