Physics benchmarks were underrating the models

A new paper argues that frontier models are far better at physics than their benchmark scores suggest, and that the fault lies with the benchmarks. The authors audited six widely used physics tests, including HLE-Physics, CMT-Benchmark, and PHYBench, then had physics experts review the cases models supposedly got wrong. Most of those wrong answers turned out to be fine, and the grading, not the model's reasoning, was at fault.

Once the flawed items are corrected, the scores jump. GPT-5.6-Sol goes from 47.3% to 78.7% on HLE-Physics and from 61.0% to 87.2% on CMT-Benchmark, and reaches 94.4% pass@4 on the retained CritPt problems. The authors read this as near-saturation on closed-ended physics questions, which means the headroom left on these tests is mostly grading error rather than genuine model failure.

The uncomfortable implication is about trust in evaluation itself. If expert review can move a headline score by more than thirty points, then a lot of published "models struggle at X" claims may be measuring bad answer keys instead of bad models. The paper makes the case that hard technical benchmarks now need expert validation of the ground truth, not just more questions.

Why it matters

If you make decisions from benchmark numbers, this is a direct warning: a low score can mean a broken test as easily as a weak model. On saturated domains like physics, the score you trust should come with an audit of how the answers were graded.