Automation bias is older than large language models. It was named in cockpit research in the 1990s, documented in hospitals in the 2000s, and it describes a simple failure: when a machine offers an answer, people tend to accept it without checking, even when the information to check it is in front of them. AI assistants that produce fluent, confident text are a near-perfect trigger for this bias, and the human-factors literature is the best guide we have to what happens next.
This article defines the terms, summarises the established evidence from aviation and medicine, and draws out what reduces the effect. It is not about sycophancy (the model agreeing with you) or cognitive offloading (outsourcing memory to a tool). It is about believing a confident answer because it was confident.
Two related failures
The NASA aviation researchers who coined the term defined automation bias by its consequences. The term refers to omission and commission errors resulting from the use of automated cues as a shortcut for vigilant information seeking. A commission error is following an automated recommendation that is wrong. An omission error is failing to notice a problem because the automation did not flag it.
A related concept is automation complacency. In the integrative review by Parasuraman and Manzey, automation complacency occurs under conditions of multiple-task load, when manual tasks compete with the automated task for the operator’s attention. Complacency is about not monitoring; bias is about accepting. The same review is blunt about how the bias shows up: automation bias results in making both omission and commission errors when decision aids are imperfect.
The key phrase is “when decision aids are imperfect”. If the automation were always right, trusting it would be rational. The bias is costly precisely because AI systems are sometimes wrong and sound the same either way.
Aviation: where it was first measured
The original studies put professional pilots in simulators with automated aids that occasionally gave wrong cues. Pilots followed the wrong cues at rates that surprised the researchers, and failed to act on problems the automation did not announce.
The most useful finding from that programme concerned accountability. Across experiments with students and commercial pilots, participants who saw themselves as accountable for how they used the automation were significantly more likely to verify its correctness, and committed significantly fewer automation-related errors. Simply feeling responsible for the outcome, rather than for pressing the button, changed behaviour.
Medicine: decision support that introduces new errors
Clinical decision support systems were adopted to reduce error, and on average they do. But a systematic review of automation bias in healthcare and related fields found that the same systems introduce errors of their own. The review identified the conditions that make bias worse: environmental mediators included workload, task complexity, and time constraint, which pressurized cognitive resources.
A second systematic review, focused on the relationship between automation bias and how hard the output is to check, found that bias was not confined to busy multitasking environments, as earlier theory held. Its conclusion is directly relevant to anyone reading AI output at a desk: AB appears to be associated with the degree of cognitive load experienced in decision tasks. The harder the answer is to verify, the more likely people are to skip verifying it.
The most striking recent demonstration is in radiology. In a 2023 experiment, 27 radiologists read 50 mammograms with a purported AI assistant that sometimes suggested the wrong category. Accuracy collapsed when the AI was wrong, and experience did not protect readers: very experienced radiologists went from a mean of 82.3% correct when the AI was right to 45.5% when it was wrong. Inexperienced readers fell further. Nobody was immune.
How this applies to AI assistants and coding tools
The systems in the studies above were narrow decision aids: a flag, a category, a recommendation. Modern AI assistants produce complete answers, explanations and code, in confident prose, on any topic. Every factor the reviews identified as increasing automation bias is present.
Confident output invites less verification. The medical review linked bias to verification complexity: when checking is expensive, people check less. A paragraph of fluent reasoning is expensive to check line by line, and its fluency is not evidence of correctness.
Time pressure and workload increase bias. Developers use coding assistants precisely when they are busy. The conditions under which people are most likely to paste a suggested function without reading it are the conditions the literature says produce the most commission errors.
Experience is not a shield. The radiology result is the clearest evidence: fifteen-plus years of expertise did not stop experienced readers from following a wrong suggestion. Senior engineers reviewing AI-generated code should assume the same about themselves.
A caveat is due. Most of the evidence here is from older, narrower systems, and research on automation bias in modern AI-driven work is still limited. Direct, well-controlled studies of automation bias with today’s coding assistants are few, and we do not cite one here. The transfer of findings is a reasoned inference, not a measured result.
Who is most at risk
The reviews point to a consistent profile: high workload, high time pressure, complex tasks, and output that is hard to verify. Add trust that has been reinforced by a run of correct answers, which is exactly what a good assistant produces, and you have the conditions in which the next wrong answer sails through.
Inexperience raises the risk further, as the mammography data showed, but experience lowers it only partially.
What reduces it
The literature is thinner on cures than on causes, but three interventions recur.
Accountability. The aviation studies found that people who felt personally responsible for outcomes verified more and erred less. In a team, this means the person who ships the code owns it, not the tool that suggested it.
Reducing verification cost. If bias tracks cognitive load and verification complexity, then anything that makes checking cheaper reduces bias: tests that run automatically, diffs small enough to read, and prompts that ask for the reasoning in a form you can audit.
Keeping the human genuinely in the loop. The reviews are consistent that training and instructions alone do not remove automation bias. What helps is a workflow in which the human’s own judgement is required before the automated output takes effect, rather than a rubber stamp after it.
Common questions
What is automation bias?
Automation bias is the tendency to accept automated output as a substitute for checking. It produces commission errors (following a wrong recommendation) and omission errors (missing a problem the automation did not flag).
Is automation bias the same as sycophancy?
No. Sycophancy is the model agreeing with you. Automation bias is you agreeing with the model, without verifying, because the output was presented confidently. The two can compound each other, but the fix for automation bias is on the human side.
Does experience protect against automation bias?
Only partly. In a 2023 study, very experienced radiologists fell from 82.3% to 45.5% accuracy when a purported AI gave a wrong suggestion; inexperienced readers fell further still.
How do you reduce automation bias when using AI?
The best-supported lever is accountability: people who feel responsible for the outcome verify more and err less. Beyond that, make verification cheap and keep a real decision point for the human before AI output is acted on.
The bottom line
People accept confident automated answers without checking, and they do it more when busy, when the task is complex, and when checking is hard. AI assistants create all three conditions at once. The evidence for what helps is older and narrower than the tools we now use, but it points one way: own the output, make checking cheap, and never let “the model said so” stand in for having looked.
Did this answer your question?
Thanks. That helps us decide what to fix.
Leave a comment