55.2% of correct AI diagnoses came after a dangerous step
In hospital tests, 55.2% of AI paths with a correct last diagnosis had a decision with moderate or dangerous possible injury.
Researchers audited AI systems for clinical use at each step of the care of a patient, and not only at the last diagnosis. They used 300 cases from 3 sources, with a maximum of 7 AI systems for each source. They found that a correct last answer does not show a safe path. The paper also tests the result when a person corrects the AI.
How it works
The audit replays the admissions of patients one checkpoint at a time. The AI sees only the data that was there at that time. Each test, treatment and release decision gets a score for possible injury and for clinical fit. An AI judge gives a score to each decision, and physicians examine a sample.
The numbers
Of the paths with a correct last diagnosis, 55.2% had a step with moderate or dangerous risk. Structured corrections fixed 73.6% of first errors, against 24.8% when the person only said no. But new dangerous decisions followed 30.2% of corrections and 19.7% of the no replies.
What is not known
The rates are different for each data source. The study replays data of admissions and does not test live patients.
Insight: A correction can fix an error and add a new risk
A related study found that one incorrect risk warning sent agents to the attacker target in 60.4% of tests. Here, structured corrections fixed more errors but also caused new dangerous steps more frequently. Feedback works in 2 directions. Each person or tool that corrects an AI also changes its risk. Test the repair, not only the answer.
Sources
Posted