The top AI agent completes 46% of clinical tasks in a new test
In many tasks, the agent wrote the correct step in its report but did not do the step in the patient record.
Researchers gave 12 AI models 100 clinical tasks in a simulated electronic health record, the system that holds patient records. The top model, GPT-5.5, completed 46% of the tasks in one try. It completed only 28% of them in all 3 of its runs. The researchers report that the top agents complete less than 50% of the work.
How it works
PhysicianBench gives an agent a task, tools and a test record. The agent does the task. Then scripts check the changes in the record. The 100 tasks come from consultations between primary care doctors and specialists.
The numbers
- The top open-source model, DeepSeek V4-Pro, completed 19%.
- Some task steps in the record did not work. In 62% of these for GPT-5.5, and 73% for Claude Opus 4.6, the report was correct but the step was not done.
- Weak clinical reasoning caused about 50% of the checks that the agents did not complete.
What is not known
The tasks use made-up patient records in a test system. How the agents do in a hospital is not known. In MedHELM, a test from 2024, the models had low results on administration tasks.
Insight: Tests now check the work, not only the text
A different test, Taste-Bench, gives a model 2 alternatives in a long task. The top model got 59.7%, and random selection gets 25%. Here, agents wrote correct steps but did not do the work. The group that makes tests which check the changed system, not the text, sets the standard. A supplier that shows only text scores does not show that the agent does the work.
Sources
- Measuring What Medical AI Can Actually DoStanford HAI
Posted
