Debate helps human judges but not weak AI model judges

Researchers did a test of AI Safety via Debate, a method to control AI systems that are more capable than persons. Each of 2 AI systems speaks for an opposite position, and a judge selects the correct one. The judge is a person or a weak model. Debate increased the precision of human judges. For weak models it gave no more precision in most settings, and it could decrease it. The paper gives no figures.

How it works

The researchers did tests with some protocols. They changed the number and the type of judges. They also did tests of voting and group deliberation. A baseline had one expert speak for one answer.

The results

• Debate increased the precision of human judges. The baseline with one expert did not.
• Weighted voting gave the top precision for groups of persons.
• For groups of language models, no method gave an important result, also with more judges.

What is not known

When all sides are equal, incorrect positions could have more effect than correct positions. The authors tell that this is a sign, not a proof. They tell of a risk for fully automatic training and control systems when an error has a large effect.

Insight: AI agreement is a weak check

In a test of 20 October 2026, 37% of GPT-4o agents changed their answers with no peer information. Here, debate gives weak AI judges no more precision. This reads as proof that a vote or a debate between AI systems is a weak check. Keep a person in the loop for important decisions.