The best open AI image detector scores 75.0% and the worst 37.5%

Researchers tested 16 open detection methods, in 23 versions, that find AI images. They used the detectors as released, with no more training. The test used 12 datasets with 2.6 million images from 291 generators. The best detector got 75.0% mean accuracy, and the worst got 37.5%. No detector is best on all datasets.

How it works

The test is zero-shot. This means each detector works as released on images it did not see in training. It is the usual case for a company that uses an open detector.

The numbers

• Flux Dev, Firefly v4 and Midjourney v7 defeat most detectors, with 18 to 30% mean accuracy.
• Detectors of the same family are different by 20 to 60% when their training data are different.
• The detector rank changes much from one dataset to the next.

What is not known

The authors find three failure patterns and give guidelines for use. The abstract does not give them. The test covers open detectors only, so it does not show how paid detectors do.

Insight: A detector knows only the generators it saw

The test shows that accuracy depends on how correct the training data are for the generator. The newest commercial generators get the lowest results. The detector seems to follow the generator, so a published result gives a small guide to the images that you must check. The authors say to select a detector for your threat. Test it on a sample of your images.