Medical AI's cost-benefit dilemma
Add Axios as your preferred source to
see more of our stories on Google.
Illustration: Lindsey Bailey/Axios
There's a basic question hanging over the clinical AI tools that are already flooding the health care system: Who exactly are they working for?
Why it matters: We're approaching a world where AI may make the kinds of decisions that clinicians have been responsible for, without ethical concerns or other factors inherent to human decision-making.
Driving the news: Bob Wachter, chair of the Department of Medicine at the University of California, San Francisco, outlined the stakes in a Substack post titled "Healthcare's Alignment Problem: A Preexisting Condition."
- Decision support tools will be saddled with the same problem that human providers face: There's rarely one "right" decision in medicine, and choosing a course of action is heavily dependent on weighing factors such as cost against what's best for the patient.
- "If the machine is spring-loaded to optimize care for every individual, we've put our entire system, indeed our society, on a path toward bankruptcy. But make cost-effectiveness the default setting, and 'AI death panels' will reenter the public discourse within a week," Wachter writes.
Between the lines: The widespread deployment of AI-enabled tools means algorithms could decide whether to conduct an early diagnostic MRI or prescribe an expensive drug with limited benefit.
- Whoever designs the decision-making process and defines the interests at stake will wield enormous power.
- This dynamic already exists for the chatbots that patients regularly use for health care guidance, which Wachter says are built "to be sticky" and maintain user engagement without necessarily being optimized for a patient's well-being.
- The solution Wachter proposes is transparency: Tools can give clinicians multiple recommendations and lay out the tradeoffs of each.
Zoom in: The question of who AI is working for is already playing out in a couple of high-profile cases involving health insurers.
- In one case, Medicare Advantage patients and their families are suing UnitedHealth for using an AI tool called nH Predict in lieu of physicians to predict when post-acute rehabilitation or skilled nursing care should end, sometimes with dire consequences. UHC has denied any use of nH Predict in coverage decisions.
- A similar case against Humana also takes up charges that a predictive tool imposed a standard recovery time on patients whose actual recovery takes longer.
- Past research has also shown how a common hospital algorithm built around optimizing spending could pass over high-risk Black patients with more chronic diseases who needed more health services. In that case, developers worked with researchers to address racial bias in the model.
The intrigue: Legal experts say traditional malpractice and product liability laws aren't practical for instances in which AI systems fail the patient, because a model's performance can vary by facility and the algorithms can be opaque.
- Some experts have suggested that hospitals should be responsible for vetting the performance of AI models they provide, though the nature of the tools often leaves facilities without the information needed to effectively reduce possible injuries.
The bottom line: No matter how much thoughtful discussion takes place, these are fraught choices. Expect heated political debate and expensive litigation over the next few years.
