Automation Bias in the use of Large Language Models

This week is AWS Re:Invent 2023 in Las Vegas, which is AWS' yearly cloud computing conference. They have lots of new products to announce, and no surprise, Generative AI and ML related tools are taking center stage. In the day 2 keynote presentation 3M Health Information Systems | M*Modal got a few shout-outs for our use of AWS Sagemaker, which is always fun to see 😁. The ability to quickly spin up clusters of accelerated computing resources for model training and inference has been a game changer for our work.

The presentations by AWS this week make it very clear that generative AI and Large Language Models are being pursued by a huge number of companies and industries. Seeing the potential of Generative AI to revolutionize work across industries is causing companies to scramble to avoid being left behind. With this rapid and widespread adoption of Generative AI, I think it's a relevant time to discuss something that's not new but is more relevant now than ever: automation bias.

As Wikipedia states, automation bias is "is the propensity for humans to favor suggestions from automated decision-making systems and to ignore contradictory information made without automation, even if it is correct." Back in 1998, researchers at NASA and the University of Chicago published an interesting paper about the appearance of automation bias in flight cockpits. The experiment explores whether the presence of an automated system makes a pilot more likely to ignore several other cues that would indicate that the automated system is responding in error. They conclude:

"Pilots who reported a higher internalized sense of accountability for their interactions with automation verified correct automation functioning more often and committed fewer errors than other pilots. These results suggest that the sense that one is accountable for one's interaction with automation encourages vigilance, proactive strategies, and the use of all information in interactions with automated systems.".

This statement is worth reading carefully to make sure we understand the implication. The trust we have of an automated system has an impact on how diligently we verify the accuracy of the system. In this study, the automation systems were designed to help monitor and control information about running an airplane. In other words, the automation systems were designed to optimize the performance of the aircraft, they weren't designed to show the pilot what they wanted to see, but to show him what he needed to see.

In terms of automation bias, we can now view Generative AI through the lens of that study. Methods for training LLMs like Reinforcement Learning from Human Feedback (RLHF) and Direct Preference Optimization (DPO) are specifically designed to align an LLM that was trained on a massive amount of data to human preferences. Although the systems are generally intended to produce correct information, first and foremost they're being fine-tuned (aligned) to produce information that is believable. The goal is to produce a believable response that is correct, but even if ChatGPT/GPT-4/Bard/Llama is completely fabricating the information, it is generally still highly believable. This is a problem because as the study on pilot automation systems stated, the amount that we verify the accuracy of the system is dependent on how much we trust the system, and LLMs are implicitly designed to fool us into trusting them! Remember back in 2022 when a google engineer believed that a chatbot was sentient? Since the output from these LLMs are generally correct for most things, we risk being lulled into complete trust of the system because we'll have an even harder time identifying the little errors that might sneak in with all of the truth.

We are starting to see this automation bias surface in several domains that are using LLMs. Of immediate interest to those working in healthcare, researchers found that radiologists were prone to automation bias when using an AI system to assist in reading mammograms. In the legal domain, some New York lawyers are being sanctioned because they used ChatGPT to generate a legal briefing that had "six fictitious case citations". The law firm's response to this sanction is most likely something we will see many people and companies repeat as LLM adoption grows:

"Levidow, Levidow & Oberman said in a statement on Thursday that its lawyers 'respectfully' disagreed with the court that they acted in bad faith. 'We made a good faith mistake in failing to believe that a piece of technology could be making up cases out of whole cloth,' the firm's statement said."

They most likely used ChatGPT a few times, were convinced by its believability, misconstrued it for truthfulness, and then allowed their trust of the system to blind them to inaccuracies that lurked among the believable output.

The world of academia is plagued with automation bias when academics choose to use an LLM to generate large swaths of content without verifying the content. One example (out of so so many that a simple google search pulled up) is researchers who used ChatGPT to generate research about an insect's (millipede) negative effects on crops that was not only incorrect, but contained completely hallucinated references.

It is exciting to see the speed at which companies like AWS are equipping companies large and small with the tools to implement Generative AI into all industries, and hopefully as these implementations expand in usage and availability, we will all continue to put critical thought into how these systems can be deployed in a way that prevents automation bias.

Since my work is concentrated in the healthcare domain, this topic serves as an important reminder to use caution in the way we train models and implement the applications that use them. We need to not only deliver information that is as truthful as possible, but need to also do it in such a way that highlights information that may be believable yet incorrect. The stakes of providing incorrect information can be too high to get wrong, and we owe it to our users to help them see where they may be falling prey to a dangerous automation bias.