When AI models refuse to fight hackers
Add Axios as your preferred source to
see more of our stories on Google.
Illustration: Maura Losch/Axios.
In the race to fend off AI-enabled cyber threats, security teams are confronting a new problem: The AI models they rely on to investigate attacks can suddenly refuse to help.
Why it matters: While AI giants race to build AI tools that can respond quickly to hacking or other malicious online threats, strict guardrails on their most powerful models lock out defenders mid-investigation, leaving them scrambling for alternatives at critical moments.
What's happening: OpenCode, an open-source AI coding agent, said it was compromised over the weekend, allowing attackers to obtain API keys belonging to some customers, including keys for OpenAI, Anthropic and Google services.
- OpenCode said the breach was "related to a large scale distillation attack that impacted 1.4% of OpenCode Console users."
- OpenCode co-creator Dax Raad said attackers had been targeting its systems continuously for 48 hours, forcing the company to keep its own AI agents running around the clock to respond.
- OpenAI, Google and Anthropic have said they worked with affected customers to revoke exposed API keys and address unauthorized usage charges.
Zoom in: Raad said on X that frontier AI models initially helped investigate the breach but then "started refusing."
- The models' safety restrictions reportedly began blocking requests as the investigation turned to potentially malicious code, even though OpenCode was trying to defend its own systems.
- OpenCode then gained access to OpenAI's Daybreak Blue program, which gives vetted cybersecurity defenders access to models with fewer restrictions.
- "We definitely need to be more proactive and we've made so many improvements over the last few days," Raad wrote on X Monday.
The big picture: The incident underscored a dilemma facing cybersecurity teams trying to figure out when to use powerful proprietary AI models, which come with built-in restrictions, or open-weight alternatives that offer greater control over how they're deployed.
- When Hugging Face discovered suspicious activity on its systems that it later linked to OpenAI's own agents, its security team turned to a Chinese open-weight model to help investigate after encountering restrictions on frontier models.
Yes, but: AI companies maintain that they need to gate access to their most powerful models to ensure that malicious actors can't abuse their cyber capabilities.
Between the lines: Cybersecurity teams will have to figure out how to balance the need for faster AI-powered defenses with restrictions designed to keep those same capabilities out of attackers' hands, Chris Jones, chief technology officer at Nightwing, told Axios.
- "We're all going to have to figure out how that race gets modulated over time," Jones said. "That's a big piece here."
Go deeper: Anthropic and OpenAI are now cybersecurity's kingmakers
