What matters in AI.

Subscribe

Research

One change to a Claim Card changes 1 in 3 of the agent verdicts

The test used 900 runs of a GPT-5.4 agent on 60 claims with expert labels.

Researchers made the Claim Card, a record of 9 fields that shows how an agent reads a claim. A change to one field changed the verdict on about 1 in 3 of the claims. One example is the threshold for data. At random, the rate is about 1 in 7.

Verdict changed after one edit Changing one field of an AI agent's interpretation changed its verdict on about 1 in 3 claims, against 1 in 7 expected by chance. 60 claims, 5 runs each. VERDICT CHANGED AFTER ONE EDIT 1 in 3 claims changed verdict One field edited Expected by chance 1 IN 7 p < 0.001 for each edit 60 claims · 5 runs each
Changing a single interpretation field, such as a stricter support threshold, changed the agent's verdict on about a third of 60 expert-labeled claims.

Sources

  1. Same Claim, Different Interpretations: How LLM Agents Resolve Ambiguity in Scientific Claimsarxiv.org
AI MATTER · NEWS · AI MATTER · NEWS ·15 OCT2026

Posted

Tags

Learn the terms in this story

More in Research

All Research news