GeneBench-Pro shows AI still stumbles on real biology analysis

OpenAI released GeneBench-Pro, a benchmark that tests whether a model can carry a real computational biology analysis from a messy dataset to a correct answer. Each of the 129 problems hands the model the kind of data a scientist actually receives from a lab or a hospital records system, along with a short prompt that gives the experimental context and defines what has to be estimated. The questions span cancer biology, pharmacogenomics, microbial genomics, and clinical translation, and each one needs several linked steps of statistical reasoning rather than a single lookup.

The scores are low, and that is the point. OpenAI's strongest model, GPT-5.6 Sol, passes 28.7 percent of the problems at its highest reasoning setting, rising to 31.5 percent with Pro mode. Claude Opus 4.8 reaches 16 percent. The progress is real even so: when the original GeneBench was built, the best frontier model scored under 5 percent, and GPT-5.6 Sol now solves close to six times as many problems as GPT-5.2 while using about two-thirds of the tokens.

What the models get wrong is more interesting than the headline number. OpenAI describes a consistent gap between noticing and acting. A model will flag a problem in the data, a batch effect or a confounder, then fail to carry that observation into the next decision, so it picks the wrong estimator or keeps going down an analysis path that looked reasonable at the start.

Why it matters

If you are hoping to hand data analysis to an agent, this is a concrete map of where it still breaks: not in writing code or recalling methods, but in following its own findings through to the right statistical choice. Check that step yourself before you trust the output.