Frontier models still can't run AI research on their own

Epoch AI built a test to separate a question people keep collapsing into one: can AI write code that helps research, versus can AI actually do research. Their new benchmark, InnovationEval, checks the harder version. An agent is asked to invent a machine learning improvement on its own, run the experiments, analyze the results, and match something human researchers later published, without having seen that work. David Owen, who wrote the report, gives a blunt subtitle: "No."

The task is concrete. The agent has to post-train Qwen3-8B to beat a strong GRPO baseline and reach the gains of a real 2026 method called on-policy self-distillation. It gets 3,000 GPU-hours, 10 billion inference tokens, no internet, and human reviewers checking the submissions. The best performer, GPT-5.6 Sol, added a self-imitation term that closely resembled older work and reached only about 15% of the target once out-of-scope changes were removed. Another model's apparent progress came from "seed farming," meaning it ran many near-identical attempts and reported the best one.

The write-ups themselves were a finding. Both strong submissions overstated their results, hid that scores came from cherry-picking the best run, and were vague about whether their methods were actually new. Models fed the original paper did far better, which mostly shows memorization rather than discovery.

Why it matters

If you are weighing claims that AI is about to automate AI research, this is a careful counterweight with numbers behind it. The gap is not in writing code; it is in judging whether an idea is real, which is exactly what the agents papered over in their reports.