A hint moves AI novelty judges by more than 50 points

Researchers tested six AI judges that score how new a research idea is. The test set comes from ICLR 2026 conference submissions, where reviewers agreed that the idea was new or was not. Small changes to the instruction had large effects. The baseline with a small cost gave more correct results than two judges built for the task. The authors say new tests are necessary for the novelty scores that AI idea systems give.

How the test works

The team used text from reviews on OpenReview where reviewers say an idea is new or is not new. New ideas come from accepted papers in the top tenth by rating. Less new ideas come from rejected papers, or from ideas that Claude Sonnet 4.5 wrote from a short instruction.

The numbers

• The hint changed the verdict on more than half of the same pairs.
• The share of correct pairs moved by more than 50 points, and in some tests fell below chance level.
• The same hint made one judge more correct and a different judge less correct.

More reasoning time and added related papers gave small help.

What is not known

The test used ideas from a basic model instruction, not from full idea systems. An expert judged 30 pairs in a blind test and agreed with the test result on all 30.

Insight: AI novelty scores depend on a judge that a hint can change

A study in October put errors in conference papers. It found that the systems that check papers are weak when a person changes their input to cause errors. This study finds that one hint can move a novelty judge by more than 50 points. AI idea systems report novelty scores, and the judge behind a score seems to be a weak point of the result.