AI systems intended to help people make tough choices — like prescribing the right drug or setting the length of a prison sentence — can instead end up effectively making those choices for them, thanks to human faith in machines.

How it works: These programs generally offer new information or a few options meant to help a human decision-maker choose more wisely. But an overworked or overly trusting person can fall into a rubber-stamping role, unquestioningly following algorithmic advice.

Why it matters: Over-reliance on potentially faulty AI can harm the people whose lives are shaped by critical decisions about employment, health care, legal proceedings and more.

The big picture: This phenomenon is called automation bias. Early studies focused on autopilot for airplanes — but as automation technology becomes more complex, the problem could get much worse with more dangerous consequences.

  • AI carries an aura of legitimacy and accuracy, burnished by overeager marketing departments and underinformed users.
  • But AI is just fancy math. Like any equation, if you give it incorrect inputs, it will return wrong answers. And if it learns patterns that don't reflect the real world, its output will be equally flawed.

Automation bias caused by simpler technologies has already been blamed for real-world disasters.

  • In 2016, a patient was prescribed the wrong medication when a pharmacist chose a similarly named drug from a list on a computer. A nurse noticed — but administered the meds anyway, assuming the electronic record was correct. The patient had heart and blood pressure problems as a result.
  • In 2010, a pipeline dumped nearly 1 million gallons of crude oil into Michigan wetlands and rivers after operators repeatedly ignored "critical alarms." They were desensitized because of previous false alarms, according to a 2016 post-mortem report — showing another threat from over-reliance on machines.

"When people have to make decisions in relatively short timeframes, with little information — this is when people will tend to just trust whatever the algorithm gives them," says Ryan Kennedy, a University of Houston professor who researches trust and automation.

  • "The worst-case scenario is somebody taking these algorithmic recommendations, not understanding them, and putting us in a life or death situation," Kennedy tells Axios.

Now, institutions are pushing AI systems further into high-stakes decisions.

  • In hospitals: A forthcoming study found that Stanford physicians "followed the advice of [an AI] model even when it was pretty clearly wrong in some cases," says Matthew Lungren, a study author and the associate director of the university's Center for Artificial Intelligence in Medicine and Imaging.
  • At war: Weapons are increasingly automated, but usually still require human approval before they shoot to kill. In a 2004 paper, Missy Cummings, now the director of Duke University's Humans and Autonomy Lab, wrote that automated aids for aviation or defense "can cause new errors in the operation of a system if not designed with human cognitive limitations in mind."
  • On the road: Sophisticated driver assists like Tesla's Autopilot still require people to intervene in dangerous situations. But a 2015 Duke study found that humans lose focus when they're just monitoring a car rather than driving it.

And in the courtroom, human prejudice mixes in.

What's next: More information about an algorithm's confidence level can give people clues for how much they should lean on it. Lungren says the Stanford physicians made fewer mistakes when they were given a recommendation and an accuracy estimate.

  • In the future, machines may adjust to a user's behavior — say, by showing its work when a person is trusting its advice too much, or by backing off if the user seems tried or stressed, which can make them less critical.
  • "Humans are good at seeing nuance in a situation that automation can't," says Neera Jain, a Purdue professor who studies human–machine interaction. "[We are] trying to avoid those situations where we become so over-reliant that we forget we have our own brains that are powerful and sophisticated."

Go deeper

1 hour ago - Sports

Disney announces partnership and documentary series with Colin Kaepernick

Photo: Carmen Mandato/Getty Images

The Walt Disney Company announced Monday that ESPN Films will produce an exclusive docuseries on political activist and former NFL player Colin Kaepernick as part of a larger deal with Kaepernick's production arm RA Vision Media.

Driving the news: Former ESPN personality Jemele Hill tweeted that she'll be serving as a producer on the docuseries, after leaving the network two years ago following a dramatic falling out in 2018. At the time, Hill's outspoken tweets about President Trump put the network in the crosshairs of a polarizing debate over race and politics.

Updated 2 hours ago - Politics & Policy

Coronavirus dashboard

Illustration: Eniola Odetunde/Axios

  1. Global: Total confirmed cases as of 12 p.m. ET: 11,495,412 — Total deaths: 535,185 — Total recoveries — 6,217,763Map.
  2. U.S.: Total confirmed cases as of 12 p.m. ET: 2,897,613 — Total deaths: 129,953 — Total recoveries: 906,763 — Total tested: 35,512,916Map.
  3. Public health: Case growth outpacing testing in hotspots — Medical community urges public to wear masks.
  4. States: Texas hospitals in danger of being overwhelmed amid surge.
  5. Politics: Meadows says Trump "is right" to claim 99% of coronavirus cases are "harmless."

Court orders temporary shutdown of Dakota Access Pipeline

Protesters against the Dakota Access Pipeline in San Francisco in 2017. Photo: Joel Angel Juarez/Anadolu Agency/Getty Images

A federal judge ordered Monday the shutdown of the Dakota Access Pipeline — a project at the heart of battles over oil-and-gas infrastructure — while the U.S. Army Corps of Engineers conducts a new environmental analysis.

Why it matters: The latest twist in the years-long fight over the pipeline is a defeat for the White House agenda of advancing fossil fuel projects and a win for Native Americans and environmentalists who oppose the project