What matters in AI.

Subscribe

55.2% of correct AI diagnoses came after a dangerous step

In hospital tests, 55.2% of AI paths with a correct last diagnosis had a decision with moderate or dangerous possible injury.

Researchers audited AI systems for clinical use at each step of the care of a patient, and not only at the last diagnosis. They used 300 cases from 3 sources, with a maximum of 7 AI systems for each source. They found that a correct last answer does not show a safe path. The paper also tests the result when a person corrects the AI.

How it works

The audit replays the admissions of patients one checkpoint at a time. The AI sees only the data that was there at that time. Each test, treatment and release decision gets a score for possible injury and for clinical fit. An AI judge gives a score to each decision, and physicians examine a sample.

The numbers

Of the paths with a correct last diagnosis, 55.2% had a step with moderate or dangerous risk. Structured corrections fixed 73.6% of first errors, against 24.8% when the person only said no. But new dangerous decisions followed 30.2% of corrections and 19.7% of the no replies.

What is not known

The rates are different for each data source. The study replays data of admissions and does not test live patients.

Insight: A correction can fix an error and add a new risk

A related study found that one incorrect risk warning sent agents to the attacker target in 60.4% of tests. Here, structured corrections fixed more errors but also caused new dangerous steps more frequently. Feedback works in 2 directions. Each person or tool that corrects an AI also changes its risk. Test the repair, not only the answer.

Sources

  1. Right Diagnosis, Unsafe Route: Auditing Clinical AI Decision Trajectories and Corrective Oversight for Validity and Potential Harmarxiv.org
AI MATTER · NEWS · AI MATTER · NEWS ·19 NOV2026

Posted

Tags

More in Research

All Research news