Can we trust LLMs to judge whether a research idea is novel?

Why it matters

Ai2 has released a study, benchmark, data, code, and an automated pipeline examining whether large language models can reliably judge the novelty of scientific research ideas. The context is Ai2's broader effort to automate parts of scientific discovery through projects like DataVoyager, AutoDiscovery, Theorizer, and Asta. As those systems begin generating research ideas, a second model, called a novelty judge, is often asked to decide whether an idea is genuinely new or repackages existing work. Ai2 found this judgment to be unstable. Testing several frontier models from OpenAI and Anthropic, the researchers showed that small changes to prompts, reasoning budgets, or evaluation formats produced dramatically different verdicts on the same ideas. Adding literature retrieval or extra reasoning did not consistently help, and purpose-built novelty evaluation systems underperformed a simple prompted baseline. The benchmark was constructed from ICLR 2026 peer reviews, mining passages where reviewers explicitly discussed originality to build pools of higher- and lower-novelty human ideas, then pairing them with AI-generated counterparts. The findings matter because automated scientific discovery depends on trustworthy novelty assessment, which this work shows remains unresolved.