No one ethics policy gives top results for all medical AI models

Researchers made MedConstBench, a benchmark (a standard test) in which the ethics policy of the model, its constitution, is the part that changes, when most older work tests only facts. They did tests on 10 open-source models with 4 constitutions. The effect is different for each model and each value, and no constitution is top for all.

How it works

The test uses the 4 principles of medical ethics of Beauchamp. It includes MedDilemma, a set of clinical dilemmas that the authors wrote. A group of models gives the scores, and persons calibrate them.

The numbers

The authors make a do-no-harm selector, which selects a constitution for each model. It gives the same result as the top selection for each model and keeps a worst-case floor (a minimum result). They say it works for new models at low cost.

What is not known

The text gives no scores and no names of the 10 models. It does not tell how the selector works with models that are not open-source.

Insight: One policy text can fail on the next model

MedConstBench shows that no policy is top for all models. The researchers of TypedBench found that a hosted model follows its policy, but its results change with the wording. The policy must be set for each model, and a change of model can remove a safety rule that no person sees. Test the policy and the model as one pair.