Illustration: Aïda Amer/Axios
One of the oddest ways that an AI system can fail is by falling prey to an adversarial attack — a cleverly manipulated input that makes the system behave in an unexpected way.
Why it matters: Autonomous car experts worry that their cameras are susceptible to these tricks: It's been shown that a few plain stickers can make a stop sign look like a "Speed Limit 100" marker to a driverless vehicle. But other high-stakes fields — like medicine — are paying too little attention to this risk.
That's according to a powerhouse of researchers from Harvard and MIT, who published an today article in Science arguing that these attacks could blindside hospitals, pharma companies, and big insurers.
Details: Consider a photo of a mole on a patient's skin. Research has shown that it can be manipulated in a way that's invisible to the human eye, but still changes the result of an AI system's diagnosis from cancerous to non-cancerous.
The big question: Why would anyone want to do this?
- For Samuel Finlayson, an MD–PhD candidate at Harvard and MIT and the lead author of the new paper, it’s a question of incentives. If someone sending in data for analysis has a different goal than the owner of the system doing the analysis, there's a potential for funny business.
- We're not talking about a malicious doctor manipulating cancer diagnoses — "There's way more effective ways to kill a person," Finlayson says — but rather an extension of existing dynamics into a near future where AI is involved in billing, diagnosis, and reading medical scans.
Doctors and hospitals already game the insurance billing system — these could be considered proto-adversarial attacks, Finlayson tells Axios.
- They often bill for more expensive procedures than they performed, in order to make more money, or avoid billing for procedures that they know will land a huge bill in the patient's lap.
- Insurance companies are already hiring tech firms to put a stop to the practice, often with AI tools. Finlayson sees a future where basic adversarial attacks are used to fool the AI systems into continuing to accept fraudulent claims.
- Despite this possibility, hospitals and the pharma industry are flying blind, he says. "Adversarial attacks aren't even on the map for them."
But, but, but: These hypotheticals are a bit far-fetched for Matthew Lungren, associate director of the Stanford Center for Artificial Intelligence in Medicine and Imaging.
- "There are a lot of easier ways to defraud the system, frankly," he tells Axios.
- But there is an urgent need, Lungren says, to test medical AI systems more rigorously before they're released into the world. Protecting against adversarial attacks is one of the ways experts should shore up algorithms before using them on patients.
Go deeper: Scientists call for rules on evaluating predictive AI in medicine