Can we trust LLMs to judge whether a research idea is novel?
Ai2 releases a study, benchmark and code showing that LLMs judging the novelty of research ideas give unstable verdicts, with small prompt changes shifting accuracy by over 50 points in one test. Retrieval and extra reasoning did not reliably improve results.
What the article says
On October 27, 2026, Ai2 released Old Ideas, Novel Problems, a study of how reliably LLMs can judge the novelty of research ideas. The release includes a paper, a benchmark, data, code and an automated pipeline for building new evaluation sets. Ai2 notes that these novelty judges are often validated only on human-written papers, even though they are increasingly used to assess AI-generated ideas.
The benchmark draws on ICLR 2026 peer reviews, with 154 high-novelty human ideas and 145 lower-novelty ones. Across six judges tested, small changes in prompt wording shifted verdicts sharply: GPT-5.4's accuracy on mixed human and generated pairs moved from 93.5% to 40.9%. More search and reasoning did not reliably help, and two purpose-built evaluators underperformed a simple prompted Claude Opus 4.6 baseline. Ai2 concludes that LLM-based novelty evaluation remains an open problem; no product changes.
Read the official article AnnuIA profile
Summary written by AnnuIA in its own words from the official page of Hugging Face. The content and rights belong to the publisher: please read the original article.