A rerun of an AI judge changed the verdict on 21.2% of prompts

Researchers used GPT-4.1 as a judge. A judge is a model that selects one of two answers as the best. They used 500 Arena-Hard v2.0 prompts and did not change the answers of two models. When they operated the default setup again, the result changed on 21.2% of prompts. The aggregate score moved only about one point.

How it works

The researchers operated the judge with eight combinations of criteria, output format and task framing. They also did each setup again with no change, at temperature zero (the adjustment for minimum random output), on a pinned model version.

The numbers

• One setup told the judge to give a verdict with no reasoning. Results changed on 28.2 percentage points more prompts than in reruns, and the score fell 4.09 points.
• A setup with precision first changed results on 6.7 percentage points more prompts. The score barely moved.

What is not known

The paper tests one judge model and one set of prompts. It does not tell if other judges work the same. This is the result of one paper.

Insight: A judge score can look stable while its verdicts change

Laurie Voss of Arize tells teams to score agent records with AI judges. This paper shows that a judge changes its verdict on 21.2% of prompts when it operates again. This reads as a weak point. A team that reports only an average score does not show the noise. Find the agreement on each prompt. If you use judge verdicts as labels, you must know this number.