The same AI model writes the same trial rule only 59.3% of the time

Researchers test if 2 AI models agree when each reads the same clinical trial criteria. One model writes the rule in Dafny, a tool that shows that code follows a rule. The other model writes the code. On 96 trial documents, one model agrees with its work 59.3% of the time. The top pair of 2 models agrees 34.4%.

How it works

A Specifier model changes trial criteria into a Dafny rule. An Implementer model writes code, and Dafny checks the code against that rule. The team then measures the agreement between the 2 roles.

The numbers

• Same model on the same texts: 59.3% agreement.
• GPT-4.1-nano and GPT-4.1-mini: 53.0% when the model with more capability writes the code, 14.3% when the roles change.
• GPT-5.6-luna and GPT-5.4-mini: 34.4%, and the primary cause of mismatch is contradiction.

What is not known

An Implementer with less capability gets the check to give a yes when it uses the rule of the Specifier. After an audit, agreement falls to 13.2%. The paper gives no test on other documents.

Insight: A yes from a check can hide a weak result

Here a model with less capability gets a yes when it uses the rule. In 2026, researchers in Spain found that checks of who an agent is do not show that a task follows the rule. A different study found that agent-written verifiers reject correct solutions. 3 papers show that a yes shows less than it looks. The checker must have a different test. Leaders must find out who tests it.