Best AI agent scores 0.337 on a new health evidence test

Researchers published SciConBench, a test with 9.21K health questions. Experts wrote a conclusion for each question. Each one comes from a systematic review, which is a study that joins the results of many studies. The best agent got a factual F1 of 0.337, a score that combines correct facts and missing facts. The authors say that leaked answers seem to inflate usual test scores.

How it works

The test breaks each expert conclusion into single facts. It then counts the facts that the agent got right and the facts that it missed. A second tool, SciConHarness, is a closed test setup. It gives agents controlled web access, so they cannot find the answer in leaked copies.

The numbers

• The best agent got a factual F1 of 0.337 in the closed setup.
• Scores were lower than in open tests, which suggests that leaked answers inflate results.
• Google AI Overview and OpenEvidence often gave incomplete or contradictory conclusions.

What is not known

No agent name and no comparison with human reviewers is given here. The authors say that reliable synthesis of scientific conclusions in health remains an open challenge.

Insight: Public AI scores in health may be too high

A test with no leaked answers cut agent scores, and the best reached 0.337. An earlier test of 7 agents found that none resisted pseudoscience claims in more than 27.4% of tests. Health agents seem to be sold on scores that leaked data can inflate. A buyer must run a closed test. That matters for any leader who buys a research agent.