Automation bias

How it shows up

Automation bias produces two errors, named in the aviation studies where the term began.

• Commission: doing what the system recommends when it is wrong.
• Omission: failing to act because the system did not prompt you.

In a 1999 study by Linda Skitka, Kathleen Mosier and Mark Burdick, people ran a simulated flight task with or without a computer aid that monitored the aircraft and recommended decisions. The aid was very reliable but not perfect. Those without it did better on the monitoring task. Those with it missed events the aid did not flag, and followed its advice even when their training and other valid indicators said otherwise.

Why it matters

Putting a person between an AI system and a decision only helps if the person still checks. Automation bias leads people to accept wrong output instead.

A 2012 systematic review pooled four healthcare studies in what it called “a small, indicative meta-analysis”. The tools studied gave clinicians text advice without interrupting them, and the studies measured errors of commission. When the tool’s advice was wrong, users were 26% more likely to make an incorrect decision than a control group without the tool.

The EU AI Act writes the risk into law. Under Article 14(4)(b), a high-risk AI system must enable the people assigned to human oversight, as appropriate and proportionate, to remain aware of automation bias. The Act stresses systems that give information or recommendations for decisions people make.

NIST’s Generative AI Profile warns that automation bias can compound confabulation. OWASP’s human agent trust exploitation entry says it helps make abuse by an AI agent “look legitimate and hard to spot”.

What reduces it

The 2012 review, drawing on 74 studies, lists mitigators: training, making users accountable for decisions, where the advice sits on the screen, confidence levels that update with the output, and giving information rather than a recommendation.

The evidence on training and accountability is mixed. Training may help users spot wrong advice, but the review also cites a study implying it had no impact. It finds training reduces errors from complacency, too little monitoring of automation, more clearly than automation bias. Parasuraman and Manzey’s 2010 review reports that automation bias “cannot be prevented by training or instructions”. On accountability, two studies found that making users accountable reduced automation bias. A third found that only users’ own sense of being accountable did.

Where definitions disagree

Automation bias and automation complacency were long studied separately. The 2012 review notes that the literature uses several synonyms for automation bias, and finds no agreed definition of complacency.

Parasuraman and Manzey’s 2010 review says complacency arises when manual tasks compete with an automated one for attention. It argues the two are “different manifestations of overlapping automation-induced phenomena”, with attention central to both.

Questions and answers

What is an example of automation bias?

In a 1999 simulated flight study by Skitka, Mosier and Burdick, people with an automated aid did what it recommended even when that contradicted their training and other valid indicators, and missed events the aid did not prompt them about. People without the aid did better on the monitoring task.

What is the difference between automation bias and automation complacency?

A 2012 systematic review draws the line at multitasking. Complacency is paying too little attention to automation because other, manual tasks take priority. Automation bias can occur without multitasking, as an active bias toward the system's advice when deciding. The review found no consensus on how to define complacency, and Parasuraman and Manzey's 2010 review treats the two as overlapping.

Does the EU AI Act mention automation bias?

Yes. Under Article 14(4)(b), a high-risk AI system must enable the people overseeing it, as appropriate and proportionate, "to remain aware of the possible tendency of automatically relying or over-relying on the output", which the Act calls automation bias. It stresses systems that give information or recommendations for decisions people make.