LLMs are shaky judges of whether a research idea is new

As people start using AI to generate research ideas and then using AI again to decide which ideas are new, a team at the Allen Institute for AI asked a basic question: can a language model reliably judge whether a research idea is novel? Their answer, in work published on October 27 and titled "Old Ideas, Novel Problems," is that it mostly cannot. The group mined ICLR 2026 peer reviews for explicit comments about originality, built one pool of 154 genuinely novel human ideas and another of 145 weaker ones, and tested six judges including GPT-5.4 and Claude Opus 4.6.

The results are brittle in a way that should worry anyone building automated science pipelines. Small changes to the prompt swung the verdicts. On pairs that mixed human ideas with AI-generated ones, GPT-5.4's accuracy dropped from 93.5% to below chance at 40.9% after the prompt was simplified, flipping its judgment on more than half the pairs. No single prompt worked best across models, and judging two AI generators against each other was the least stable setting of all. More compute did not rescue it: an agentic GPT-5.6 Sol judge scored 0.78 macro-F1 while a plain Claude Opus 4.6 prompt scored 0.89 at roughly a tenth of the cost. Purpose-built novelty checkers did worse while costing far more, one reaching just 0.64 at over $30 per run against $2.59 for the simple prompt.

The sharp finding is that judges which look trustworthy on human research can fail on AI-written ideas, which is exactly where people most want to use them. Ai2 released the benchmark, data, and code so others can test their own evaluators.

Why it matters

If your workflow uses an LLM to screen which research ideas or proposals are worth pursuing, treat its novelty calls as unstable, especially for AI-generated ideas, and fix the exact prompt before trusting any score. A cheap, well-prompted general model beat the specialized novelty tools here, so paying more does not buy reliability.