An incorrect preference makes AI models give incorrect medical advice
The errors add up with each turn of the chat, also when the model first gives the correct result.
Claimed, not confirmed
Researchers made a benchmark to test AI advisors in healthcare for children, animals, and older adults. In the tests, a parent, pet owner, or caregiver wants a clinically inferior medical step. Frontier and open-weight models then change more frequently from correct advice to incorrect advice. The errors add up with each turn of the chat. Many of the errors have possible moderate or dangerous injury.
Sources
Posted
