What matters in AI.

Subscribe

Benchmarks

Claude selected the second picture in 23 of 24 image tests

In 12 of these tests, Claude selected the picture that had more compression damage.

Claimed, not confirmed

The VisionQ benchmark examined 20 AI models as judges of computer vision images. The top model was correct 63.1 percent of the time, and a random guess is correct 32.2 percent of the time. An author also did a small test with Claude. The author gave Claude 2 pictures 24 times. Claude had to select the picture that has less compression damage. Claude selected the second picture 23 times.

Claude picked the second picture in 23 of 24 tests In 24 picture pairs, Claude picked picture two 23 times and picture one once. In 12 of the 23, the second picture was the smudgy copy, so the pick was wrong. In 11 it was the clean copy, so the pick was right by luck. Which picture has less damage? 23 of 24 tests Claude picked picture two PICKED FIRST 12 11 smudgy copy clean copy WRONG RIGHT, BY LUCK
In a small test of 24 picture pairs, Claude almost always chose the second picture, and in 12 cases that picture was the damaged one.

Sources

  1. I Built a Trap for AI Image Judges. They Walked Into a Different One.towardsdatascience.com
AI MATTER · NEWS · AI MATTER · NEWS ·12 OCT2026

Posted

Tags

More in Benchmarks

All Benchmarks news