Stanford finds 44 percent of 50 AI model claims are not correct

Owen Queen and James Zou of Stanford tested 50 known claims about large language models (LLMs, AI that writes text). They used 20 models from eight developers, released from July 2024 to September 2026. Of the claims, 44 percent are not correct at this time, because the first effect has disappeared for most of them. The 22 results with an end date stayed correct for 15.7 months on average.

How it works

The team tested first results again on new models. Claims about tasks that models cannot do stop to be correct first. The researchers think that these errors have one correct answer, and developers can train the models to it. Claims about bias, and about models that agree with users, stay correct for more time.

The numbers

• Citations that a model made up: 55 percent for GPT-3.5 in 2023, 0 percent for GPT-6.1 Sol.
• Qwen3.7-Flash, also a 2026 model, continues to make up 69 percent.
• Claude Sonnet 5.5 gives an incorrect answer in about 10 percent of questions with 2 known facts.

What is not known

Results are different between models. Release date, provider and model size do not tell which results stop to be correct. The researchers tell authors to give the model, version and date, and to give prompts and code.

Insight: Test scores expire fast, so buyers need their own test

A Japanese test found that six models check primary claims less correctly than side claims. Stanford finds that the model, not the vendor or the size, sets the result. Google tests a sleep phase that changes a model after release. This reads as a move from the score of the vendor to the test of the buyer. Buyers who keep a test set and operate it again at each model change get the control.