Six LLMs check primary answer claims less correctly than side claims

Researchers made JSFactCheckBench, a Japanese benchmark that tests how correctly LLMs check facts claim by claim. It has 1,913 claims from LLM answers to questions about misleading information on Japanese social media. Across six LLMs, all check claims that answer the question less correctly than supplementary claims. The aggregate scores hide this difference. The test covers claims about misleading information from Japanese social media.

How it works

Each claim has a label for factuality at a reference time and a label for importance to the question. The proof comes from preselected Web sources. The benchmark records the conditions behind each label: which questions, how each claim is judged and which sources count.

The numbers

• Scores drop when models must find the related text in longer or multiple sources.
• For all four open models, scores drop when the question and response are added.
• The source gives no score values.

What is not known

The test covers Japanese claims and six LLMs. The authors tell that other benchmarks do not tell the conditions, and they cannot show these weak points. This is the result of one paper.

Insight: The checker fails most on the claims that matter most

An earlier test of AI tools that find logic errors in papers found them weak against adversarial changes. This paper finds that claim checkers are weaker on the claims that answer the question. The game looks like a hidden selection. The person who sets the test conditions sets the score, and a high score can hide the weak point. Leaders who use AI to check facts must tell vendors to give scores by claim type.