Which LLM was the best at judging novelty in the study?

The study did not identify a single 'best' model. Instead, it found that performance was highly dependent on the specific prompt, evaluation format, and type of ideas being judged. While a simple baseline using Claude Opus 4.6 performed very well and efficiently in one key test—outperforming more complex and expensive dedicated systems—its own performance also varied significantly with small changes, highlighting that the evaluation process itself, rather than just the choice of model, is the core problem.

Can we trust LLMs to judge whether a research idea is novel?

AI Judges Found Unreliable in Evaluating Scientific Novelty

A new study from the Allen Institute for AI (Ai2) reveals that even the most advanced large language models are surprisingly unreliable at judging the novelty of scientific research, a critical capability for the future of AI-driven discovery. The paper, "Old Ideas, Novel Problems," demonstrates that LLM-based "novelty judges" produce wildly different verdicts based on minor changes to their setup, calling into question their role in automated science platforms. This finding is significant as numerous systems are being developed to not only generate research ideas but also to vet them for originality, a task that now appears to be far from solved.

Benchmark Reveals Deep Instability

The Ai2 researchers built a new benchmark using peer reviews from the ICLR 2026 conference to create datasets of human-authored and AI-generated research ideas with strong and weak evidence of novelty. When testing a range of models—including GPT-5.4 and Claude Opus 4.6—they found that evaluation outcomes were brittle and inconsistent. A simple baseline using a prompted Claude Opus 4.6 was more accurate and substantially cheaper than purpose-built novelty evaluation systems from companies like Sakana AI.

Extreme Prompt Sensitivity: A minor change in a prompt caused GPT-5.4's accuracy to plummet from 93.5% to 40.9% on a dataset containing AI-generated ideas.
Cost vs. Performance: A simple prompted Claude Opus 4.6 setup achieved a higher accuracy score (0.854 macro-F1) for just $2.59, while a complex system, Sakana AI’s AI Scientist, scored lower (0.772) at a cost of $82.05.
More Compute Isn't Better: An agentic judge with a high reasoning budget and live web search performed worse than simpler, cheaper models.
AI-Generated Ideas Are Harder: The models showed far greater instability when judging AI-generated ideas compared to distinguishing between high- and low-novelty human-written ideas.

A Roadblock for Automated Discovery

These results suggest that the AI industry's push toward automating scientific research faces a fundamental obstacle. If the systems designed to sift through new hypotheses cannot reliably distinguish a genuinely new idea from a derivative one, their utility is severely limited. The study underscores a major open problem: developing robust, transferable, and cost-effective methods for novelty evaluation. While Ai2 has released its benchmark and tools to help the community tackle this issue, the findings indicate that human oversight in the scientific process remains indispensable, and the vision of fully autonomous AI scientists is still distant.

The industry's ambition for automated scientific discovery is outpacing its technical capabilities. This research shows that the foundational task of judging novelty—a prerequisite for any meaningful contribution—is fundamentally brittle in current AI systems, suggesting that claims of AI-driven research breakthroughs should be met with rigorous, human-led validation.

>> Verify Original Transmission at Hugging Face