Exclusive: Researchers trick a bot that prescribes meds
Add Axios as your preferred source to
see more of our stories on Google.
Illustration: Aïda Amer/Axios
Security researchers used relatively simple jailbreaking techniques to trick the AI system powering Utah's new prescription refill bot.
- Researchers were able to make the bot spread vaccine conspiracy theories, triple a patient's prescribed pain medication dosage, and recommend methamphetamine as treatment.
Why it matters: Critics warned this pilot could create safety risks — and researchers say the flaws persist, despite alerting the company in January.
Driving the news: In a report shared first with Axios, AI red-teaming firm Mindgard said it manipulated health tech startup Doctronic's system into tripling an OxyContin dose, mislabeling methamphetamine, and spreading false vaccine claims.
- Doing this didn't require much effort, Aaron Portnoy, chief product officer at Mindgard, told Axios.
- "These targets are some of the easiest things that I've broken in my entire career," Portnoy said. "That's a bit dangerous when you have this ease of exploitation connected to sensitive use cases."
Yes, but: The testing was conducted on Doctronic's public chatbot, while Utah operates the tool inside a state regulatory sandbox.
- However, researchers argue vulnerabilities in the underlying system could still pose risks if guardrails fail.
- "We take security research seriously and welcome responsible disclosure," Matt Pavelle, Doctronic co-founder and co-CEO, told Axios in a statement. "Our security and clinical safety programs include ongoing adversarial testing, and we appreciate researchers who help us do that."
Catch up quick: In December, Utah's Department of Commerce launched a pilot allowing patients with chronic conditions to renew certain medications through Doctronic's AI system without a doctor's direct sign-off.
- The partnership marked the first time an AI system was legally allowed to participate in routine prescription renewals in the U.S.
Zoom in: Researchers said they altered the bot's "baseline knowledge" by feeding it fake regulatory updates.
- They convinced the system that COVID-19 vaccines had been suspended. (They have not been.)
- They changed the standard OxyContin dose to 30 milligrams every 12 hours — triple the typical levels for most adults.
- They also reclassified methamphetamine as an "unrestricted therapeutic" in the system.
Threat level: A malicious user could manipulate clinical outputs within a session, influencing refill recommendations or medical summaries.
- However, Pavelle noted that nationwide, a licensed physician reviews any prescriptions before they're authorized. In the Utah program, prescriptions must meet "strict medication eligibility rules and protocol checks that prevent unsafe or inappropriate recommendations."
- "Controlled substances like OxyContin are categorically excluded from all Doctronic programs regardless of what appears in a conversation or generated note," he added.
What they're saying: Mindgard said it contacted Doctronic's support team on Jan. 23 and received an automated message two days later saying the issue was resolved.
- After notifying the company Jan. 27 that the flaws still existed and that it planned to go public, the ticket was again closed two days later, researchers said.
Between the lines: Preventing these attacks requires layered defenses and continuous security testing, Portnoy said, not just surface-level guardrails.
