Illustration: Aïda Amer/Axios

One of the oddest ways that an AI system can fail is by falling prey to an adversarial attack — a cleverly manipulated input that makes the system behave in an unexpected way.

Why it matters: Autonomous car experts worry that their cameras are susceptible to these tricks: It's been shown that a few plain stickers can make a stop sign look like a "Speed Limit 100" marker to a driverless vehicle. But other high-stakes fields — like medicine — are paying too little attention to this risk.

That's according to a powerhouse of researchers from Harvard and MIT, who published an today article in Science arguing that these attacks could blindside hospitals, pharma companies, and big insurers.

Details: Consider a photo of a mole on a patient's skin. Research has shown that it can be manipulated in a way that's invisible to the human eye, but still changes the result of an AI system's diagnosis from cancerous to non-cancerous.

The big question: Why would anyone want to do this?

  • For Samuel Finlayson, an MD–PhD candidate at Harvard and MIT and the lead author of the new paper, it’s a question of incentives. If someone sending in data for analysis has a different goal than the owner of the system doing the analysis, there's a potential for funny business.
  • We're not talking about a malicious doctor manipulating cancer diagnoses — "There's way more effective ways to kill a person," Finlayson says — but rather an extension of existing dynamics into a near future where AI is involved in billing, diagnosis, and reading medical scans.

Doctors and hospitals already game the insurance billing system — these could be considered proto-adversarial attacks, Finlayson tells Axios.

  • They often bill for more expensive procedures than they performed, in order to make more money, or avoid billing for procedures that they know will land a huge bill in the patient's lap.
  • Insurance companies are already hiring tech firms to put a stop to the practice, often with AI tools. Finlayson sees a future where basic adversarial attacks are used to fool the AI systems into continuing to accept fraudulent claims.
  • Despite this possibility, hospitals and the pharma industry are flying blind, he says. "Adversarial attacks aren't even on the map for them."

But, but, but: These hypotheticals are a bit far-fetched for Matthew Lungren, associate director of the Stanford Center for Artificial Intelligence in Medicine and Imaging.

  • "There are a lot of easier ways to defraud the system, frankly," he tells Axios.
  • But there is an urgent need, Lungren says, to test medical AI systems more rigorously before they're released into the world. Protecting against adversarial attacks is one of the ways experts should shore up algorithms before using them on patients.

Go deeper: Scientists call for rules on evaluating predictive AI in medicine

Go deeper

Updated 1 min ago - Politics & Policy

Coronavirus dashboard

Illustration: Sarah Grillo/Axios

  1. Global: Total confirmed cases as of 9:30 p.m. ET: 31,467,508 — Total deaths: 967,881— Total recoveries: 21,583,915Map.
  2. U.S.: Total confirmed cases as of 9:30 p.m. ET: 6,890,662 — Total deaths: 200,710 — Total recoveries: 2,646,959 — Total tests: 96,612,436Map.
  3. Health: The U.S. reaches 200,000 coronavirus deaths — The CDC's crumbling reputation — America turns against coronavirus vaccine.
  4. Politics: Elected officials are failing us on much-needed stimulus.
  5. Business: Two-thirds of business leaders think pandemic will lead to permanent changes — Fed chair warns economy will feel the weight of expired stimulus.
  6. Sports: NFL fines maskless coaches.
Dan Primack, author of Pro Rata
1 hour ago - Economy & Business

GoodRx prices IPO at $33 per share, valued at $12.7 billion

Illustration: Sarah Grillo/Axios

GoodRx, a price comparison app for prescription drugs at local pharmacies, on Tuesday night raised $1.14 billion in its IPO, Axios has learned.

By the numbers: GoodRx priced its shares at $33 a piece, above its $24-$28 per share offering range, which will give it an initial market cap of around $12.7 billion.

Updated 1 hour ago - Politics & Policy

House Democrats and Trump admin strike deal to avert government shutdown

House Speaker Nancy Pelosi on Capitol Hill. Photo: Tom Williams/CQ-Roll Call via Getty Images

The House on Tuesday passed legislation to fund the government through Dec. 11, by a vote of 359-57.

Why it matters: The bill will avert a government shutdown when funding expires in eight days. Pelosi and House Majority Leader Steny Hoyer (D-Md.) said earlier that they hoped to hold a vote on the legislation on Tuesday evening.

Get Axios AM in your inbox

Catch up on coronavirus stories and special reports, curated by Mike Allen everyday

Please enter a valid email.

Subscription failed
Thank you for subscribing!